Parent Log:
http://ci.aztec-labs.com/2476e3d34226c110
Command: d8033ca60b1dba76 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/9ddf7d35886489afed34aca3cb819b32c86194b4
Env: REF_NAME=gh-readonly-queue/next/pr-15215-99e234fe4473771b32124e0402885610a6a3f98a CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 24 09:23:04 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15215_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
09:23:09 [09:23:09.215]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f5f99ec169ededd epochNumber=42
{"provingJobId":"1:BASE_PARITY:2f5f99ec169ededd"}
09:23:09 [09:23:09.226]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:5d3f42b3a58c7e62 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5d3f42b3a58c7e62"}
09:23:09 [09:23:09.259]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.260]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43c15d0448a409ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:43c15d0448a409ed"}
09:23:09 [09:23:09.262]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c75aafab9076f16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c75aafab9076f16"}
09:23:09 [09:23:09.264]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8675f80d67f22b47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8675f80d67f22b47"}
09:23:09 [09:23:09.267]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8675f80d67f22b47 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8675f80d67f22b47"}
09:23:09 [09:23:09.270]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.271]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43948d87e3c83706 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43948d87e3c83706"}
09:23:09 [09:23:09.272]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:43948d87e3c83706 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:43948d87e3c83706"}
09:23:09 [09:23:09.273]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:43948d87e3c83706 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:43948d87e3c83706"}
09:23:09 [09:23:09.274]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:43948d87e3c83706 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:43948d87e3c83706"}
09:23:09 [09:23:09.276]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e97797e6bd8c36b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e97797e6bd8c36b9"}
09:23:09 [09:23:09.298]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.300]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.301]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a76502a99d3a5254 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a76502a99d3a5254"}
09:23:09 [09:23:09.301]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a76502a99d3a5254
{"provingJobId":"1:BASE_PARITY:a76502a99d3a5254"}
09:23:09 [09:23:09.302]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a76502a99d3a5254 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a76502a99d3a5254"}
09:23:09 [09:23:09.302]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a76502a99d3a5254 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a76502a99d3a5254"}
09:23:09 [09:23:09.303]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.303]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d06f4baeeb37b6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d06f4baeeb37b6c"}
09:23:09 [09:23:09.304]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0d06f4baeeb37b6c
{"provingJobId":"1:BASE_PARITY:0d06f4baeeb37b6c"}
09:23:09 [09:23:09.304]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0d06f4baeeb37b6c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:0d06f4baeeb37b6c"}
09:23:09 [09:23:09.305]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.306]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11022c6d35baedd9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:11022c6d35baedd9"}
09:23:09 [09:23:09.307]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:11022c6d35baedd9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:11022c6d35baedd9"}
09:23:09 [09:23:09.307]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:11022c6d35baedd9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:11022c6d35baedd9"}
09:23:09 [09:23:09.308]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.309]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a39ff2c8859c9eed epochNumber=1
{"provingJobId":"1:BASE_PARITY:a39ff2c8859c9eed"}
09:23:09 [09:23:09.309]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a39ff2c8859c9eed type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a39ff2c8859c9eed"}
09:23:09 [09:23:09.309]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a39ff2c8859c9eed type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a39ff2c8859c9eed"}
09:23:09 [09:23:09.310]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.311]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f654883b0142c00 epochNumber=0
{"provingJobId":"1:BASE_PARITY:4f654883b0142c00"}
09:23:09 [09:23:09.311]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4f654883b0142c00 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4f654883b0142c00"}
09:23:09 [09:23:09.311]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e203baccb7fd49fe epochNumber=0
{"provingJobId":"1:BASE_PARITY:e203baccb7fd49fe"}
09:23:09 [09:23:09.312]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4f654883b0142c00 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4f654883b0142c00"}
09:23:09 [09:23:09.312]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf867f335245436b epochNumber=0
{"provingJobId":"1:BASE_PARITY:cf867f335245436b"}
09:23:09 [09:23:09.313]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.325]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.326]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48cffe640f36db57 epochNumber=2
{"provingJobId":"1:BASE_PARITY:48cffe640f36db57"}
09:23:09 [09:23:09.326]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8382dbf33001f6c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8382dbf33001f6c4"}
09:23:09 [09:23:09.328]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.328]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95f0b4281bed4fcc epochNumber=1
{"provingJobId":"1:BASE_PARITY:95f0b4281bed4fcc"}
09:23:09 [09:23:09.329]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.330]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e7cd2dd50f59e37 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e7cd2dd50f59e37"}
09:23:09 [09:23:09.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1cf94054ede63335 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1cf94054ede63335"}
09:23:09 [09:23:09.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2119f46313a0da43 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2119f46313a0da43"}
09:23:09 [09:23:09.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b865b7c6010252c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b865b7c6010252c0"}
09:23:09 [09:23:09.332]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.333]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9b9ad2303812034 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c9b9ad2303812034"}
09:23:09 [09:23:09.333]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:738eeff6f9e2fe55 epochNumber=1
{"provingJobId":"1:BASE_PARITY:738eeff6f9e2fe55"}
09:23:09 [09:23:09.334]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d172e4aa5bdea703 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d172e4aa5bdea703"}
09:23:09 [09:23:09.334]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1881f2b385f8883f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1881f2b385f8883f"}
09:23:09 [09:23:09.335]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.336]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cdc628071a77fc6f epochNumber=1
{"provingJobId":"1:BASE_PARITY:cdc628071a77fc6f"}
09:23:09 [09:23:09.336]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:719b095771dc2dd3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:719b095771dc2dd3"}
09:23:09 [09:23:09.336]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26d8974cf9a97d8f epochNumber=2
{"provingJobId":"1:BASE_PARITY:26d8974cf9a97d8f"}
09:23:09 [09:23:09.337]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d5a01947a300846 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d5a01947a300846"}
09:23:09 [09:23:09.338]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.339]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19a40e44b342e6a3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:19a40e44b342e6a3"}
09:23:09 [09:23:09.339]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72d7a79827373bdc epochNumber=1
{"provingJobId":"1:BASE_PARITY:72d7a79827373bdc"}
09:23:09 [09:23:09.339]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:19a40e44b342e6a3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:19a40e44b342e6a3"}
09:23:09 [09:23:09.341]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.341]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee65d3c227270ec7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee65d3c227270ec7"}
09:23:09 [09:23:09.341]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9d12ea8af5f25fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:c9d12ea8af5f25fa"}
09:23:09 [09:23:09.342]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ee65d3c227270ec7 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:ee65d3c227270ec7"}
09:23:09 [09:23:09.343]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.343]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:18e0da2d0ae056e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:18e0da2d0ae056e9"}
09:23:09 [09:23:09.343]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f4679cbf5a6958f epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f4679cbf5a6958f"}
09:23:09 [09:23:09.344]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:18e0da2d0ae056e9 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:18e0da2d0ae056e9"}
09:23:09 [09:23:09.345]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.345]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f97af911c6ffb50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f97af911c6ffb50"}
09:23:09 [09:23:09.346]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2f97af911c6ffb50
{"provingJobId":"1:BASE_PARITY:2f97af911c6ffb50"}
09:23:09 [09:23:09.346]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2f97af911c6ffb50 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2f97af911c6ffb50"}
09:23:09 [09:23:09.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:533e0f343a881da9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:533e0f343a881da9"}
09:23:09 [09:23:09.347]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2f97af911c6ffb50 has already been completed
{"provingJobId":"1:BASE_PARITY:2f97af911c6ffb50"}
09:23:09 [09:23:09.348]
INFO:
prover-client:proving-broker Proving Broker started
09:23:09 [09:23:09.348]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5bb30a8009cd4a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5bb30a8009cd4a7"}
09:23:09 [09:23:09.349]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:503cb2ccc0c60d1e epochNumber=2
{"provingJobId":"1:BASE_PARITY:503cb2ccc0c60d1e"}
09:23:10 [09:23:10.401]
INFO:
prover-client:proving-broker Proving Broker started
09:23:10 [09:23:10.402]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e5bb30a8009cd4a7 settled=false
{"provingJobId":"1:BASE_PARITY:e5bb30a8009cd4a7","status":"pending"}
09:23:10 [09:23:10.402]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:503cb2ccc0c60d1e settled=false
{"provingJobId":"1:BASE_PARITY:503cb2ccc0c60d1e","status":"pending"}
09:23:10 [09:23:10.403]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e5bb30a8009cd4a7 type=BASE_PARITY startedAt=1750756989349 older agent has taken job
{"provingJobId":"1:BASE_PARITY:e5bb30a8009cd4a7"}
09:23:10 [09:23:10.404]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e5bb30a8009cd4a7 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:e5bb30a8009cd4a7"}
09:23:10 [09:23:10.405]
INFO:
prover-client:proving-broker Proving Broker started
09:23:10 [09:23:10.406]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de321d9983522b95 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de321d9983522b95"}
09:23:10 [09:23:10.406]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c294b03280421e1e epochNumber=2
{"provingJobId":"1:BASE_PARITY:c294b03280421e1e"}
09:23:11 [09:23:11.407]
INFO:
prover-client:proving-broker Proving Broker started
09:23:11 [09:23:11.407]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:de321d9983522b95 settled=false
{"provingJobId":"1:BASE_PARITY:de321d9983522b95","status":"pending"}
09:23:11 [09:23:11.407]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c294b03280421e1e settled=false
{"provingJobId":"1:BASE_PARITY:c294b03280421e1e","status":"pending"}
09:23:11 [09:23:11.408]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:de321d9983522b95 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:de321d9983522b95"}
09:23:11 [09:23:11.409]
INFO:
prover-client:proving-broker Proving Broker started
09:23:11 [09:23:11.409]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95c6bc644fb74ad8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:95c6bc644fb74ad8"}
09:23:11 [09:23:11.410]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03813af1c84022e7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:03813af1c84022e7"}
09:23:12 [09:23:12.411]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.412]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:95c6bc644fb74ad8 settled=false
{"provingJobId":"1:BASE_PARITY:95c6bc644fb74ad8","status":"pending"}
09:23:12 [09:23:12.412]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:03813af1c84022e7 settled=false
{"provingJobId":"1:BASE_PARITY:03813af1c84022e7","status":"pending"}
09:23:12 [09:23:12.413]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:95c6bc644fb74ad8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:95c6bc644fb74ad8"}
09:23:12 [09:23:12.413]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:95c6bc644fb74ad8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:95c6bc644fb74ad8"}
09:23:12 [09:23:12.414]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.415]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f990ce0bc37a3884 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f990ce0bc37a3884"}
09:23:12 [09:23:12.415]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0685bb98ca932e0c epochNumber=2
{"provingJobId":"1:BASE_PARITY:0685bb98ca932e0c"}
09:23:12 [09:23:12.416]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f990ce0bc37a3884 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f990ce0bc37a3884"}
09:23:12 [09:23:12.416]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0685bb98ca932e0c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0685bb98ca932e0c"}
09:23:12 [09:23:12.418]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.418]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:607cbbbfa2f9a88b epochNumber=1
{"provingJobId":"1:BASE_PARITY:607cbbbfa2f9a88b"}
09:23:12 [09:23:12.419]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:486bf6f37189f403 epochNumber=2
{"provingJobId":"1:BASE_PARITY:486bf6f37189f403"}
09:23:12 [09:23:12.419]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:607cbbbfa2f9a88b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:607cbbbfa2f9a88b"}
09:23:12 [09:23:12.419]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:607cbbbfa2f9a88b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:607cbbbfa2f9a88b"}
09:23:12 [09:23:12.419]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:486bf6f37189f403 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:486bf6f37189f403"}
09:23:12 [09:23:12.421]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.421]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:9daac534e6b5b5ae err=test error
{"provingJoId":"1:BASE_PARITY:9daac534e6b5b5ae"}
09:23:12 [09:23:12.422]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.423]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d8b7fcd815712bc3 not found
{"provingJobId":"1:BASE_PARITY:d8b7fcd815712bc3"}
09:23:12 [09:23:12.425]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.425]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0dfaca91d0c2f8eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:0dfaca91d0c2f8eb"}
09:23:12 [09:23:12.427]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.427]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e938a3fcd213449a epochNumber=1
{"provingJobId":"1:BASE_PARITY:e938a3fcd213449a"}
09:23:12 [09:23:12.528]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e938a3fcd213449a timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e938a3fcd213449a"}
09:23:12 [09:23:12.536]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:992561ab12bd1f43 epochNumber=1
{"provingJobId":"1:BASE_PARITY:992561ab12bd1f43"}
09:23:12 [09:23:12.642]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:992561ab12bd1f43 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:992561ab12bd1f43"}
09:23:12 [09:23:12.644]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3441cc194a1ed098 epochNumber=10
{"provingJobId":"1:BASE_PARITY:3441cc194a1ed098"}
09:23:12 [09:23:12.668]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:12 [09:23:12.771]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:63a013b7d9dac576 epochNumber=1
{"provingJobId":"1:BASE_PARITY:63a013b7d9dac576"}
09:23:12 [09:23:12.948]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:63a013b7d9dac576 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:63a013b7d9dac576"}
09:23:12 [09:23:12.977]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.979]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:754b866669a92657 epochNumber=1
{"provingJobId":"1:BASE_PARITY:754b866669a92657"}
09:23:12 [09:23:12.980]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:754b866669a92657 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:754b866669a92657"}
09:23:12 [09:23:12.981]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.982]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2aff4ff80e79a21 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2aff4ff80e79a21"}
09:23:12 [09:23:12.983]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f2aff4ff80e79a21 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f2aff4ff80e79a21"}
09:23:12 [09:23:12.984]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f2aff4ff80e79a21 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:f2aff4ff80e79a21"}
09:23:12 [09:23:12.985]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.986]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e8c22fcb82820c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e8c22fcb82820c0"}
09:23:12 [09:23:12.986]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1e8c22fcb82820c0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1e8c22fcb82820c0"}
09:23:12 [09:23:12.987]
INFO:
prover-client:proving-broker Proving Broker started
09:23:12 [09:23:12.988]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c540f9db680577ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:c540f9db680577ff"}
09:23:12 [09:23:12.988]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c540f9db680577ff type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c540f9db680577ff"}
09:23:12 [09:23:12.989]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99d0f31976aa804b epochNumber=3
{"provingJobId":"1:BASE_PARITY:99d0f31976aa804b"}
09:23:13 [09:23:13.013]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:13 [09:23:13.015]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c540f9db680577ff err=test error
{"provingJoId":"1:BASE_PARITY:c540f9db680577ff"}
09:23:13 [09:23:13.017]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.017]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b6de47fcff6e936d settled=false
{"provingJobId":"1:BASE_PARITY:b6de47fcff6e936d","status":"pending"}
09:23:13 [09:23:13.017]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2a353aa53fbef983 settled=false
{"provingJobId":"1:BASE_PARITY:2a353aa53fbef983","status":"pending"}
09:23:13 [09:23:13.020]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.020]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bc2c69186a96df45 settled=true
{"provingJobId":"1:BASE_PARITY:bc2c69186a96df45","status":"fulfilled"}
09:23:13 [09:23:13.020]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:a44d396b3aeb5e20 settled=true
{"provingJobId":"2:BASE_PARITY:a44d396b3aeb5e20","status":"fulfilled"}
09:23:13 [09:23:13.022]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.023]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7a0d758c087f8e72 settled=true
{"provingJobId":"1:BASE_PARITY:7a0d758c087f8e72","status":"fulfilled"}
09:23:13 [09:23:13.023]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b6e23730dc5b3440 settled=false
{"provingJobId":"1:BASE_PARITY:b6e23730dc5b3440","status":"pending"}
09:23:13 [09:23:13.024]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94272ec941eaab6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:94272ec941eaab6a"}
09:23:13 [09:23:13.027]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.027]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb81540c54320c60 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb81540c54320c60"}
09:23:13 [09:23:13.028]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:fb81540c54320c60: Error: db error: Error: db error
09:23:13 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:fb81540c54320c60"}
09:23:13 [09:23:13.030]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.030]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c1dd5a443410f6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c1dd5a443410f6a"}
09:23:13 [09:23:13.031]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9c1dd5a443410f6a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9c1dd5a443410f6a"}
09:23:13 [09:23:13.035]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9c1dd5a443410f6a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9c1dd5a443410f6a"}
09:23:13 [09:23:13.037]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.038]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:444d7ea119d913b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:444d7ea119d913b3"}
09:23:13 [09:23:13.038]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:444d7ea119d913b3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:444d7ea119d913b3"}
09:23:13 [09:23:13.038]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:444d7ea119d913b3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:444d7ea119d913b3"}
09:23:13 [09:23:13.039]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:444d7ea119d913b3: Error: db error
09:23:13 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:444d7ea119d913b3"}
09:23:13 [09:23:13.040]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.041]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8eb4dcc46bc57cae epochNumber=1
{"provingJobId":"1:BASE_PARITY:8eb4dcc46bc57cae"}
09:23:13 [09:23:13.041]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8eb4dcc46bc57cae type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8eb4dcc46bc57cae"}
09:23:13 [09:23:13.042]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8eb4dcc46bc57cae type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8eb4dcc46bc57cae"}
09:23:13 [09:23:13.043]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.044]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21c77a3e6064f845 epochNumber=1
{"provingJobId":"1:BASE_PARITY:21c77a3e6064f845"}
09:23:13 [09:23:13.044]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:21c77a3e6064f845 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:21c77a3e6064f845"}
09:23:13 [09:23:13.045]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:21c77a3e6064f845 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:21c77a3e6064f845"}
09:23:13 [09:23:13.046]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:21c77a3e6064f845 jobErr=test error: Error: db error
09:23:13 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:21c77a3e6064f845"}
09:23:13 [09:23:13.047]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.048]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:73d99fd49bb78361 not found
{"provingJobId":"1:BASE_PARITY:73d99fd49bb78361"}
09:23:13 [09:23:13.050]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.051]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c93abd164d9b31bb err=test error
{"provingJoId":"1:BASE_PARITY:c93abd164d9b31bb"}
09:23:13 [09:23:13.052]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f733d147d86337d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f733d147d86337d0"}
09:23:13 [09:23:13.064]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f733d147d86337d0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f733d147d86337d0"}
09:23:13 [09:23:13.064]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f733d147d86337d0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f733d147d86337d0"}
09:23:13 [09:23:13.074]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:409387beeb0d45d8 epochNumber=2
{"provingJobId":"2:BASE_PARITY:409387beeb0d45d8"}
09:23:13 [09:23:13.075]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:409387beeb0d45d8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:409387beeb0d45d8"}
09:23:13 [09:23:13.075]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:409387beeb0d45d8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:409387beeb0d45d8"}
09:23:13 [09:23:13.137]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:6d99b076c03813d1 epochNumber=3
{"provingJobId":"3:BASE_PARITY:6d99b076c03813d1"}
09:23:13 [09:23:13.154]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:13 [09:23:13.199]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:fcb9ebfe55630514 epochNumber=4
{"provingJobId":"4:BASE_PARITY:fcb9ebfe55630514"}
09:23:13 [09:23:13.207]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:13 [09:23:13.260]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:3f5e82aea44df129 epochNumber=5
{"provingJobId":"5:BASE_PARITY:3f5e82aea44df129"}
09:23:13 [09:23:13.285]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:13 [09:23:13.315]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.315]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e76078b97f47d371 epochNumber=42
{"provingJobId":"1:BASE_PARITY:e76078b97f47d371"}
09:23:13 [09:23:13.317]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testIXIvMP/42 with map size 1073741824
09:23:13 [09:23:13.322]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.354]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:bf26e3c341a8b45c epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf26e3c341a8b45c"}
09:23:13 [09:23:13.360]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.361]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b651ac3c26b348c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b651ac3c26b348c2"}
09:23:13 [09:23:13.362]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDTVSMP/1 with map size 1073741824
09:23:13 [09:23:13.363]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.380]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ccd08e2fc250469 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ccd08e2fc250469"}
09:23:13 [09:23:13.392]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.392]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ccb00f8d43db5610 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ccb00f8d43db5610"}
09:23:13 [09:23:13.393]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testr6bweC/1 with map size 1073741824
09:23:13 [09:23:13.396]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.414]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ccb00f8d43db5610 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ccb00f8d43db5610"}
09:23:13 [09:23:13.417]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2aa9cb1617df5671 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2aa9cb1617df5671"}
09:23:13 [09:23:13.418]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQ4ar51/1 with map size 1073741824
09:23:13 [09:23:13.420]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.443]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2aa9cb1617df5671 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2aa9cb1617df5671"}
09:23:13 [09:23:13.444]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2aa9cb1617df5671 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2aa9cb1617df5671"}
09:23:13 [09:23:13.451]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2aa9cb1617df5671 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2aa9cb1617df5671"}
09:23:13 [09:23:13.455]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.456]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6cf44a228e3a5ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6cf44a228e3a5ed"}
09:23:13 [09:23:13.457]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-tests65D5k/1 with map size 1073741824
09:23:13 [09:23:13.459]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.479]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.482]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.483]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:315a5d41becf405b epochNumber=1
{"provingJobId":"1:BASE_PARITY:315a5d41becf405b"}
09:23:13 [09:23:13.484]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtOMjdb/1 with map size 1073741824
09:23:13 [09:23:13.486]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.507]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:315a5d41becf405b
{"provingJobId":"1:BASE_PARITY:315a5d41becf405b"}
09:23:13 [09:23:13.507]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:315a5d41becf405b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:315a5d41becf405b"}
09:23:13 [09:23:13.508]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:315a5d41becf405b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:315a5d41becf405b"}
09:23:13 [09:23:13.517]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.517]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86d683b0bf601116 epochNumber=1
{"provingJobId":"1:BASE_PARITY:86d683b0bf601116"}
09:23:13 [09:23:13.518]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0Rb0Uc/1 with map size 1073741824
09:23:13 [09:23:13.520]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.539]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:86d683b0bf601116
{"provingJobId":"1:BASE_PARITY:86d683b0bf601116"}
09:23:13 [09:23:13.540]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:86d683b0bf601116 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:86d683b0bf601116"}
09:23:13 [09:23:13.552]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.553]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19faf329b4d90354 epochNumber=1
{"provingJobId":"1:BASE_PARITY:19faf329b4d90354"}
09:23:13 [09:23:13.554]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnlWd8u/1 with map size 1073741824
09:23:13 [09:23:13.556]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.576]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:19faf329b4d90354 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:19faf329b4d90354"}
09:23:13 [09:23:13.577]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:19faf329b4d90354 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:19faf329b4d90354"}
09:23:13 [09:23:13.585]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.586]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4631c09ba6020495 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4631c09ba6020495"}
09:23:13 [09:23:13.586]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdQ9z9w/1 with map size 1073741824
09:23:13 [09:23:13.589]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.606]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4631c09ba6020495 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4631c09ba6020495"}
09:23:13 [09:23:13.607]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4631c09ba6020495 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4631c09ba6020495"}
09:23:13 [09:23:13.617]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59f435099c3892df epochNumber=0
{"provingJobId":"1:BASE_PARITY:59f435099c3892df"}
09:23:13 [09:23:13.619]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testVAIYtR/0 with map size 1073741824
09:23:13 [09:23:13.621]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.640]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:59f435099c3892df epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:59f435099c3892df"}
09:23:13 [09:23:13.641]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2570fec6afb93a5 epochNumber=0
{"provingJobId":"1:BASE_PARITY:e2570fec6afb93a5"}
09:23:13 [09:23:13.642]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:59f435099c3892df epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:59f435099c3892df"}
09:23:13 [09:23:13.642]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd934a9b53bddc07 epochNumber=0
{"provingJobId":"1:BASE_PARITY:dd934a9b53bddc07"}
09:23:13 [09:23:13.658]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.661]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.662]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90e1d93edd8f9a60 epochNumber=2
{"provingJobId":"1:BASE_PARITY:90e1d93edd8f9a60"}
09:23:13 [09:23:13.662]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testEYi3Um/2 with map size 1073741824
09:23:13 [09:23:13.664]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e68fab3298856e74 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e68fab3298856e74"}
09:23:13 [09:23:13.685]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEYi3Um/1 with map size 1073741824
09:23:13 [09:23:13.688]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.709]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.710]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d77ecd3cbe32018a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d77ecd3cbe32018a"}
09:23:13 [09:23:13.711]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjWIbFF/1 with map size 1073741824
09:23:13 [09:23:13.713]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.735]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.736]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52f4acda22c6789d epochNumber=1
{"provingJobId":"1:BASE_PARITY:52f4acda22c6789d"}
09:23:13 [09:23:13.737]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFc3Ks6/1 with map size 1073741824
09:23:13 [09:23:13.741]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.763]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20929624b2ba189d epochNumber=1
{"provingJobId":"1:BASE_PARITY:20929624b2ba189d"}
09:23:13 [09:23:13.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:298362315035bb64 epochNumber=2
{"provingJobId":"1:BASE_PARITY:298362315035bb64"}
09:23:13 [09:23:13.770]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFc3Ks6/2 with map size 1073741824
09:23:13 [09:23:13.773]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.792]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f849b9b303ee8de epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f849b9b303ee8de"}
09:23:13 [09:23:13.802]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06f0497afb6dd7ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:06f0497afb6dd7ff"}
09:23:13 [09:23:13.803]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGY6p1S/1 with map size 1073741824
09:23:13 [09:23:13.805]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.822]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ccfaf7611d380fa1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ccfaf7611d380fa1"}
09:23:13 [09:23:13.827]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0691252b75892436 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0691252b75892436"}
09:23:13 [09:23:13.828]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGY6p1S/2 with map size 1073741824
09:23:13 [09:23:13.830]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.848]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7b73e29cf923d24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7b73e29cf923d24"}
09:23:13 [09:23:13.856]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.857]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:543679be360ca523 epochNumber=1
{"provingJobId":"1:BASE_PARITY:543679be360ca523"}
09:23:13 [09:23:13.858]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testODYYst/1 with map size 1073741824
09:23:13 [09:23:13.860]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.877]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25a3335c62f3bdb3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:25a3335c62f3bdb3"}
09:23:13 [09:23:13.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92e53e416649944d epochNumber=2
{"provingJobId":"1:BASE_PARITY:92e53e416649944d"}
09:23:13 [09:23:13.885]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testODYYst/2 with map size 1073741824
09:23:13 [09:23:13.887]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.904]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5f0111cddff3b3f epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5f0111cddff3b3f"}
09:23:13 [09:23:13.913]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.914]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40c7008781190b2b epochNumber=1
{"provingJobId":"1:BASE_PARITY:40c7008781190b2b"}
09:23:13 [09:23:13.915]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRhMybC/1 with map size 1073741824
09:23:13 [09:23:13.917]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af40a098541284a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af40a098541284a6"}
09:23:13 [09:23:13.940]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:40c7008781190b2b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:40c7008781190b2b"}
09:23:13 [09:23:13.965]
INFO:
prover-client:proving-broker Proving Broker started
09:23:13 [09:23:13.966]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce92caeff392430a epochNumber=1
{"provingJobId":"1:BASE_PARITY:ce92caeff392430a"}
09:23:13 [09:23:13.966]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2FXiKW/1 with map size 1073741824
09:23:13 [09:23:13.968]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:13 [09:23:13.986]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43118b187125eff2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43118b187125eff2"}
09:23:13 [09:23:13.992]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ce92caeff392430a type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:ce92caeff392430a"}
09:23:14 [09:23:14.004]
INFO:
prover-client:proving-broker Proving Broker started
09:23:14 [09:23:14.004]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2275edd28767f318 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2275edd28767f318"}
09:23:14 [09:23:14.005]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXgVy5Y/1 with map size 1073741824
09:23:14 [09:23:14.007]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:14 [09:23:14.023]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5eecf5c7743150f7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5eecf5c7743150f7"}
09:23:14 [09:23:14.031]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2275edd28767f318 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:2275edd28767f318"}
09:23:14 [09:23:14.036]
INFO:
prover-client:proving-broker Proving Broker started
09:23:14 [09:23:14.037]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bfefe083b86d7ed6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bfefe083b86d7ed6"}
09:23:14 [09:23:14.037]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeGYltI/1 with map size 1073741824
09:23:14 [09:23:14.040]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:14 [09:23:14.058]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:bfefe083b86d7ed6
{"provingJobId":"1:BASE_PARITY:bfefe083b86d7ed6"}
09:23:14 [09:23:14.059]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bfefe083b86d7ed6 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:bfefe083b86d7ed6"}
09:23:14 [09:23:14.065]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c5e66c81db52275 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7c5e66c81db52275"}
09:23:14 [09:23:14.072]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bfefe083b86d7ed6 has already been completed
{"provingJobId":"1:BASE_PARITY:bfefe083b86d7ed6"}
09:23:14 [09:23:14.077]
INFO:
prover-client:proving-broker Proving Broker started
09:23:14 [09:23:14.078]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8acb6774637838d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8acb6774637838d"}
09:23:14 [09:23:14.079]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTWIq8p/1 with map size 1073741824
09:23:14 [09:23:14.086]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:14 [09:23:14.105]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a854e2dca8aba9a epochNumber=2
{"provingJobId":"1:BASE_PARITY:8a854e2dca8aba9a"}
09:23:14 [09:23:14.107]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTWIq8p/2 with map size 1073741824
09:23:14 [09:23:14.109]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:15 [09:23:15.177]
INFO:
prover-client:proving-broker Proving Broker started
09:23:15 [09:23:15.185]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b8acb6774637838d settled=false
{"provingJobId":"1:BASE_PARITY:b8acb6774637838d","status":"pending"}
09:23:15 [09:23:15.187]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8a854e2dca8aba9a settled=false
{"provingJobId":"1:BASE_PARITY:8a854e2dca8aba9a","status":"pending"}
09:23:15 [09:23:15.188]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b8acb6774637838d type=BASE_PARITY startedAt=1750756994126 older agent has taken job
{"provingJobId":"1:BASE_PARITY:b8acb6774637838d"}
09:23:15 [09:23:15.189]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b8acb6774637838d type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:b8acb6774637838d"}
09:23:15 [09:23:15.192]
INFO:
prover-client:proving-broker Proving Broker started
09:23:15 [09:23:15.193]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2e51afdc0f52898 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c2e51afdc0f52898"}
09:23:15 [09:23:15.194]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgepSrR/1 with map size 1073741824
09:23:15 [09:23:15.195]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:15 [09:23:15.213]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b2355e21cd2f48d epochNumber=2
{"provingJobId":"1:BASE_PARITY:4b2355e21cd2f48d"}
09:23:15 [09:23:15.215]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testgepSrR/2 with map size 1073741824
09:23:15 [09:23:15.218]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:16 [09:23:16.241]
INFO:
prover-client:proving-broker Proving Broker started
09:23:16 [09:23:16.243]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c2e51afdc0f52898 settled=false
{"provingJobId":"1:BASE_PARITY:c2e51afdc0f52898","status":"pending"}
09:23:16 [09:23:16.246]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4b2355e21cd2f48d settled=false
{"provingJobId":"1:BASE_PARITY:4b2355e21cd2f48d","status":"pending"}
09:23:16 [09:23:16.247]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c2e51afdc0f52898 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:c2e51afdc0f52898"}
09:23:16 [09:23:16.254]
INFO:
prover-client:proving-broker Proving Broker started
09:23:16 [09:23:16.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9444fc3d4e50599 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a9444fc3d4e50599"}
09:23:16 [09:23:16.256]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlojknc/1 with map size 1073741824
09:23:16 [09:23:16.260]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:16 [09:23:16.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:793544e1ce3651c7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:793544e1ce3651c7"}
09:23:16 [09:23:16.280]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlojknc/2 with map size 1073741824
09:23:16 [09:23:16.282]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:17 [09:23:17.301]
INFO:
prover-client:proving-broker Proving Broker started
09:23:17 [09:23:17.303]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a9444fc3d4e50599 settled=false
{"provingJobId":"1:BASE_PARITY:a9444fc3d4e50599","status":"pending"}
09:23:17 [09:23:17.304]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:793544e1ce3651c7 settled=false
{"provingJobId":"1:BASE_PARITY:793544e1ce3651c7","status":"pending"}
09:23:17 [09:23:17.305]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a9444fc3d4e50599 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a9444fc3d4e50599"}
09:23:17 [09:23:17.306]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a9444fc3d4e50599 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a9444fc3d4e50599"}
09:23:17 [09:23:17.315]
INFO:
prover-client:proving-broker Proving Broker started
09:23:17 [09:23:17.316]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d1ef70b4d7d29e4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d1ef70b4d7d29e4"}
09:23:17 [09:23:17.317]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhVgflt/1 with map size 1073741824
09:23:17 [09:23:17.319]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:17 [09:23:17.337]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45f8201af8f2fb69 epochNumber=2
{"provingJobId":"1:BASE_PARITY:45f8201af8f2fb69"}
09:23:17 [09:23:17.338]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testhVgflt/2 with map size 1073741824
09:23:17 [09:23:17.340]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:17 [09:23:17.362]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3d1ef70b4d7d29e4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3d1ef70b4d7d29e4"}
09:23:17 [09:23:17.369]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:45f8201af8f2fb69 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:45f8201af8f2fb69"}
09:23:17 [09:23:17.380]
INFO:
prover-client:proving-broker Proving Broker started
09:23:17 [09:23:17.381]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1a8fdfdbf8f2169 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1a8fdfdbf8f2169"}
09:23:17 [09:23:17.382]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWjPOoB/1 with map size 1073741824
09:23:17 [09:23:17.384]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:17 [09:23:17.401]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10ad72491f1d3b2e epochNumber=2
{"provingJobId":"1:BASE_PARITY:10ad72491f1d3b2e"}
09:23:17 [09:23:17.402]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWjPOoB/2 with map size 1073741824
09:23:17 [09:23:17.405]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:17 [09:23:17.423]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a1a8fdfdbf8f2169 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a1a8fdfdbf8f2169"}
09:23:17 [09:23:17.424]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a1a8fdfdbf8f2169 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a1a8fdfdbf8f2169"}
09:23:17 [09:23:17.430]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:10ad72491f1d3b2e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:10ad72491f1d3b2e"}
09:23:17 [09:23:17.442]
INFO:
prover-client:proving-broker Proving Broker started
09:23:17 [09:23:17.445]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:53284069a6bb4aec err=test error
{"provingJoId":"1:BASE_PARITY:53284069a6bb4aec"}
09:23:17 [09:23:17.452]
INFO:
prover-client:proving-broker Proving Broker started
09:23:17 [09:23:17.453]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:39a3477fcfa83aa2 not found
{"provingJobId":"1:BASE_PARITY:39a3477fcfa83aa2"}
09:23:17 [09:23:17.458]
INFO:
prover-client:proving-broker Proving Broker started
09:23:17 [09:23:17.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e12fe84e72b3c4fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:e12fe84e72b3c4fe"}
09:23:17 [09:23:17.461]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgs4bYu/1 with map size 1073741824
09:23:17 [09:23:17.464]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:17 [09:23:17.490]
INFO:
prover-client:proving-broker Proving Broker started
09:23:17 [09:23:17.492]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ce1fe4db0a92b19 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ce1fe4db0a92b19"}
09:23:17 [09:23:17.493]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9GlKdq/1 with map size 1073741824
09:23:17 [09:23:17.497]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:17 [09:23:17.619]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7ce1fe4db0a92b19 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:7ce1fe4db0a92b19"}
09:23:17 [09:23:17.625]
INFO:
prover-client:proving-broker Proving Broker started
09:23:17 [09:23:17.626]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d128791bd5c189da epochNumber=1
{"provingJobId":"1:BASE_PARITY:d128791bd5c189da"}
09:23:17 [09:23:17.627]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3LDZNs/1 with map size 1073741824
09:23:17 [09:23:17.640]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:17 [09:23:17.779]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d128791bd5c189da timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:d128791bd5c189da"}
09:23:17 [09:23:17.781]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ac9c2b04371d6eb epochNumber=10
{"provingJobId":"1:BASE_PARITY:6ac9c2b04371d6eb"}
09:23:17 [09:23:17.783]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test3LDZNs/10 with map size 1073741824
09:23:17 [09:23:17.787]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:17 [09:23:17.833]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:17 [09:23:17.836]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
09:23:17 [09:23:17.951]
INFO:
prover-client:proving-broker Proving Broker started
09:23:17 [09:23:17.953]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9dc5e257fb36073e epochNumber=1
{"provingJobId":"1:BASE_PARITY:9dc5e257fb36073e"}
09:23:17 [09:23:17.954]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhSorDS/1 with map size 1073741824
09:23:17 [09:23:17.956]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.131]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9dc5e257fb36073e timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9dc5e257fb36073e"}
09:23:18 [09:23:18.183]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.184]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a08be37cb669e019 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a08be37cb669e019"}
09:23:18 [09:23:18.185]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLTc0XM/1 with map size 1073741824
09:23:18 [09:23:18.187]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.204]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a08be37cb669e019 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a08be37cb669e019"}
09:23:18 [09:23:18.211]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.212]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10a8680cc8fde68c epochNumber=1
{"provingJobId":"1:BASE_PARITY:10a8680cc8fde68c"}
09:23:18 [09:23:18.213]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9Dzcjx/1 with map size 1073741824
09:23:18 [09:23:18.216]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.234]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:10a8680cc8fde68c type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:10a8680cc8fde68c"}
09:23:18 [09:23:18.239]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:10a8680cc8fde68c type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:10a8680cc8fde68c"}
09:23:18 [09:23:18.251]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.252]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6fabbd2b31363a4d epochNumber=1
{"provingJobId":"1:BASE_PARITY:6fabbd2b31363a4d"}
09:23:18 [09:23:18.253]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcorlZL/1 with map size 1073741824
09:23:18 [09:23:18.254]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.271]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6fabbd2b31363a4d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6fabbd2b31363a4d"}
09:23:18 [09:23:18.279]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.280]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f293801bad38c12 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7f293801bad38c12"}
09:23:18 [09:23:18.281]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test65b6jL/1 with map size 1073741824
09:23:18 [09:23:18.283]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.301]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:7f293801bad38c12 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:7f293801bad38c12"}
09:23:18 [09:23:18.302]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f65b8f45fee24c9a epochNumber=3
{"provingJobId":"1:BASE_PARITY:f65b8f45fee24c9a"}
09:23:18 [09:23:18.303]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test65b6jL/3 with map size 1073741824
09:23:18 [09:23:18.306]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.333]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:18 [09:23:18.334]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
09:23:18 [09:23:18.353]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7f293801bad38c12 err=test error
{"provingJoId":"1:BASE_PARITY:7f293801bad38c12"}
09:23:18 [09:23:18.369]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testy0B5CD/1 with map size 1073741824
09:23:18 [09:23:18.372]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.396]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testy0B5CD/2 with map size 1073741824
09:23:18 [09:23:18.398]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.417]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.419]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:13588d72c665e1a6 settled=false
{"provingJobId":"1:BASE_PARITY:13588d72c665e1a6","status":"pending"}
09:23:18 [09:23:18.422]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fc4cafa98e95eb45 settled=false
{"provingJobId":"1:BASE_PARITY:fc4cafa98e95eb45","status":"pending"}
09:23:18 [09:23:18.428]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvefume/1 with map size 1073741824
09:23:18 [09:23:18.439]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.460]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvefume/2 with map size 1073741824
09:23:18 [09:23:18.464]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.493]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.497]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0f26269d52f0567c settled=true
{"provingJobId":"1:BASE_PARITY:0f26269d52f0567c","status":"fulfilled"}
09:23:18 [09:23:18.498]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:bb3941d2039c9b1f settled=true
{"provingJobId":"2:BASE_PARITY:bb3941d2039c9b1f","status":"fulfilled"}
09:23:18 [09:23:18.506]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOWiQ4n/1 with map size 1073741824
09:23:18 [09:23:18.508]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.540]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOWiQ4n/2 with map size 1073741824
09:23:18 [09:23:18.545]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.565]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.567]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:03ee49224aee8b2a settled=true
{"provingJobId":"1:BASE_PARITY:03ee49224aee8b2a","status":"fulfilled"}
09:23:18 [09:23:18.569]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1c4cf4e0843b7bd4 settled=false
{"provingJobId":"1:BASE_PARITY:1c4cf4e0843b7bd4","status":"pending"}
09:23:18 [09:23:18.575]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.576]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c3385d3af2157a52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c3385d3af2157a52"}
09:23:18 [09:23:18.580]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpuxkHx/1 with map size 1073741824
09:23:18 [09:23:18.582]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.603]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.604]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc08fb5631af01f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc08fb5631af01f2"}
09:23:18 [09:23:18.605]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:fc08fb5631af01f2: Error: db error: Error: db error
09:23:18 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:fc08fb5631af01f2"}
09:23:18 [09:23:18.607]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f07d90c95a69475 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7f07d90c95a69475"}
09:23:18 [09:23:18.608]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfTNvjW/1 with map size 1073741824
09:23:18 [09:23:18.612]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.635]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7f07d90c95a69475 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7f07d90c95a69475"}
09:23:18 [09:23:18.635]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7f07d90c95a69475 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7f07d90c95a69475"}
09:23:18 [09:23:18.651]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.651]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c90bca0f417da80b epochNumber=1
{"provingJobId":"1:BASE_PARITY:c90bca0f417da80b"}
09:23:18 [09:23:18.654]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSSQrmx/1 with map size 1073741824
09:23:18 [09:23:18.658]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.679]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c90bca0f417da80b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c90bca0f417da80b"}
09:23:18 [09:23:18.680]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c90bca0f417da80b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c90bca0f417da80b"}
09:23:18 [09:23:18.681]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:c90bca0f417da80b: Error: db error
09:23:18 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:c90bca0f417da80b"}
09:23:18 [09:23:18.687]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.688]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a692abb12f07813 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a692abb12f07813"}
09:23:18 [09:23:18.689]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testG4DGW1/1 with map size 1073741824
09:23:18 [09:23:18.693]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.715]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3a692abb12f07813 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3a692abb12f07813"}
09:23:18 [09:23:18.715]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3a692abb12f07813 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3a692abb12f07813"}
09:23:18 [09:23:18.727]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.728]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a1301063f366026 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a1301063f366026"}
09:23:18 [09:23:18.729]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUftJED/1 with map size 1073741824
09:23:18 [09:23:18.740]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.765]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3a1301063f366026 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3a1301063f366026"}
09:23:18 [09:23:18.768]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3a1301063f366026 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3a1301063f366026"}
09:23:18 [09:23:18.769]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:3a1301063f366026 jobErr=test error: Error: db error
09:23:18 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:3a1301063f366026"}
09:23:18 [09:23:18.776]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.777]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:148f637261856603 not found
{"provingJobId":"1:BASE_PARITY:148f637261856603"}
09:23:18 [09:23:18.783]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.784]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:13777c600aabe622 err=test error
{"provingJoId":"1:BASE_PARITY:13777c600aabe622"}
09:23:18 [09:23:18.789]
INFO:
prover-client:proving-broker Proving Broker started
09:23:18 [09:23:18.800]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:617e8864df9d6e3c epochNumber=1
{"provingJobId":"1:BASE_PARITY:617e8864df9d6e3c"}
09:23:18 [09:23:18.801]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4kx8xC/1 with map size 1073741824
09:23:18 [09:23:18.807]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.827]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:617e8864df9d6e3c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:617e8864df9d6e3c"}
09:23:18 [09:23:18.828]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:617e8864df9d6e3c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:617e8864df9d6e3c"}
09:23:18 [09:23:18.847]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:0fb58cdeebef46ee epochNumber=2
{"provingJobId":"2:BASE_PARITY:0fb58cdeebef46ee"}
09:23:18 [09:23:18.849]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4kx8xC/2 with map size 1073741824
09:23:18 [09:23:18.852]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.875]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:0fb58cdeebef46ee type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:0fb58cdeebef46ee"}
09:23:18 [09:23:18.876]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:0fb58cdeebef46ee type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:0fb58cdeebef46ee"}
09:23:18 [09:23:18.943]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:0f5e7f6ce44eb078 epochNumber=3
{"provingJobId":"3:BASE_PARITY:0f5e7f6ce44eb078"}
09:23:18 [09:23:18.944]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test4kx8xC/3 with map size 1073741824
09:23:18 [09:23:18.950]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:18 [09:23:18.994]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:18 [09:23:18.995]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
09:23:19 [09:23:19.035]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:4be7aacf9dbb5a3e epochNumber=4
{"provingJobId":"4:BASE_PARITY:4be7aacf9dbb5a3e"}
09:23:19 [09:23:19.036]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-test4kx8xC/4 with map size 1073741824
09:23:19 [09:23:19.042]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:19 [09:23:19.078]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:19 [09:23:19.079]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
09:23:19 [09:23:19.129]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:6239ddcb7d2123f7 epochNumber=5
{"provingJobId":"5:BASE_PARITY:6239ddcb7d2123f7"}
09:23:19 [09:23:19.130]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-test4kx8xC/5 with map size 1073741824
09:23:19 [09:23:19.137]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:23:19 [09:23:19.184]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:23:19 [09:23:19.185]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
09:23:19
PASS src/proving_broker/proving_broker.test.ts (
13.658 s)
09:23:19 ProvingBroker
09:23:19 Producer API
09:23:19
✓ refuses stale jobs (46 ms)
09:23:19
✓ enqueues jobs (5 ms)
09:23:19
✓ ignores duplicate jobs (5 ms)
09:23:19
✓ reports correct status when enqueuing repeat jobs (6 ms)
09:23:19
✓ throws an error in case of duplicate job IDs (22 ms)
09:23:19
✓ returns not-found status for non-existing jobs (1 ms)
09:23:19
✓ cancels jobs in queue (2 ms)
09:23:19
✓ cancels jobs in-progress (2 ms)
09:23:19
✓ returns job result if successful (2 ms)
09:23:19
✓ returns job error if failed (2 ms)
09:23:19
✓ correctly returns job status for concurrent writes (3 ms)
09:23:19
✓ refuses stale jobs (45 ms)
09:23:19
✓ enqueues jobs (31 ms)
09:23:19
✓ ignores duplicate jobs (26 ms)
09:23:19
✓ reports correct status when enqueuing repeat jobs (37 ms)
09:23:19
✓ throws an error in case of duplicate job IDs (24 ms)
09:23:19
✓ returns not-found status for non-existing jobs (3 ms)
09:23:19
✓ cancels jobs in queue (34 ms)
09:23:19
✓ cancels jobs in-progress (35 ms)
09:23:19
✓ returns job result if successful (34 ms)
09:23:19
✓ returns job error if failed (31 ms)
09:23:19
✓ correctly returns job status for concurrent writes (40 ms)
09:23:19 Consumer API
09:23:19
✓ returns undefined if no jobs are available (11 ms)
09:23:19
✓ returns jobs in priority order (2 ms)
09:23:19
✓ returns undefined if no jobs are available for the given allowList (2 ms)
09:23:19
✓ returns a job if it is in the allowList (3 ms)
09:23:19
✓ returns the most important job if it is in the allowList (3 ms)
09:23:19
✓ returns any job if filter is empty (2 ms)
09:23:19
✓ returns a new job when reporting job success (2 ms)
09:23:19
✓ returns a new job when reporting permanent error (2 ms)
09:23:19
✓ returns a new job when reporting retry-able error (2 ms)
09:23:19
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
09:23:19
✓ returns a new job if job is already in progress elsewhere (1057 ms)
09:23:19
✓ avoids sending the same job to a new agent after a restart (1003 ms)
09:23:19
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
09:23:19
✓ tracks job result if in progress (3 ms)
09:23:19
✓ tracks job result even if job is in queue (3 ms)
09:23:19
✓ ignores reported job error if unknown job (2 ms)
09:23:19
✓ ignores job result if unknown job (2 ms)
09:23:19
✓ returns undefined if no jobs are available (4 ms)
09:23:19
✓ returns jobs in priority order (47 ms)
09:23:19
✓ returns undefined if no jobs are available for the given allowList (24 ms)
09:23:19
✓ returns a job if it is in the allowList (67 ms)
09:23:19
✓ returns the most important job if it is in the allowList (54 ms)
09:23:19
✓ returns any job if filter is empty (56 ms)
09:23:19
✓ returns a new job when reporting job success (40 ms)
09:23:19
✓ returns a new job when reporting permanent error (49 ms)
09:23:19
✓ returns a new job when reporting retry-able error (32 ms)
09:23:19
✓ returns a new job when reporting progress if current one is cancelled (40 ms)
09:23:19
✓ returns a new job if job is already in progress elsewhere (1116 ms)
09:23:19
✓ avoids sending the same job to a new agent after a restart (1059 ms)
09:23:19
✓ avoids sending a completed job to a new agent after a restart (1062 ms)
09:23:19
✓ tracks job result if in progress (63 ms)
09:23:19
✓ tracks job result even if job is in queue (62 ms)
09:23:19
✓ ignores reported job error if unknown job (8 ms)
09:23:19
✓ ignores job result if unknown job (8 ms)
09:23:19 Timeouts
09:23:19
✓ tracks in progress jobs (2 ms)
09:23:19
✓ re-enqueues jobs that time out (109 ms)
09:23:19
✓ cancel stale jobs that time out (235 ms)
09:23:19
✓ keeps the jobs in progress while it is alive (205 ms)
09:23:19
✓ tracks in progress jobs (32 ms)
09:23:19
✓ re-enqueues jobs that time out (135 ms)
09:23:19
✓ cancel stale jobs that time out (322 ms)
09:23:19
✓ keeps the jobs in progress while it is alive (232 ms)
09:23:19 Retries
09:23:19
✓ retries jobs (4 ms)
09:23:19
✓ retries up to a maximum number of times (3 ms)
09:23:19
✓ passing retry=false does not retry (2 ms)
09:23:19
✓ does not retry if job is stale (29 ms)
09:23:19
✓ retries jobs (25 ms)
09:23:19
✓ retries up to a maximum number of times (42 ms)
09:23:19
✓ passing retry=false does not retry (28 ms)
09:23:19
✓ does not retry if job is stale (83 ms)
09:23:19 Database management
09:23:19
✓ re-enqueues proof requests on start (3 ms)
09:23:19
✓ restores proof results on start (2 ms)
09:23:19
✓ only re-enqueues unfinished jobs (3 ms)
09:23:19
✓ saves job when enqueued (2 ms)
09:23:19
✓ does not retain job if database fails to save (2 ms)
09:23:19
✓ saves job result (7 ms)
09:23:19
✓ saves result even if database fails to save (3 ms)
09:23:19
✓ saves job error (2 ms)
09:23:19
✓ saves job error even if database fails to save (4 ms)
09:23:19
✓ does not save job result if job is unknown (2 ms)
09:23:19
✓ does not save job error if job is unknown (2 ms)
09:23:19
✓ cleans up old jobs periodically (260 ms)
09:23:19
✓ re-enqueues proof requests on start (64 ms)
09:23:19
✓ restores proof results on start (76 ms)
09:23:19
✓ only re-enqueues unfinished jobs (70 ms)
09:23:19
✓ saves job when enqueued (28 ms)
09:23:19
✓ does not retain job if database fails to save (3 ms)
09:23:19
✓ saves job result (39 ms)
09:23:19
✓ saves result even if database fails to save (39 ms)
09:23:19
✓ saves job error (40 ms)
09:23:19
✓ saves job error even if database fails to save (49 ms)
09:23:19
✓ does not save job result if job is unknown (6 ms)
09:23:19
✓ does not save job error if job is unknown (6 ms)
09:23:19
✓ cleans up old jobs periodically (427 ms)
09:23:19
09:23:19
Test Suites: 1 passed, 1 total
09:23:19
Tests: 96 passed, 96 total
09:23:19
Snapshots: 0 total
09:23:19
Time: 13.76 s
09:23:19
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
09:23:19
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?