Parent Log:
http://ci.aztec-labs.com/7ed4060013f2034a
Command: f2922c9072735d65 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/ee3655e025191bc85c5b3d3122eeedb44e38ffb2
Env: REF_NAME=gh-readonly-queue/next/pr-15076-60e60f99f2634ad75a76b6e103858ff9ccd041d5 CURRENT_VERSION=0.87.6 CI_FULL=0
Date: Mon Jun 16 19:15:13 UTC 2025
System: ARCH=arm64 CPUS=64 MEM=247Gi HOSTNAME=pr-15076_arm64_a1-fast
Resources: CPU_LIST=0-63 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
19:15:16 [19:15:16.443]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.448]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44716704a7be38bb epochNumber=42
{"provingJobId":"1:BASE_PARITY:44716704a7be38bb"}
19:15:16 [19:15:16.451]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:c08b5b5461a73e03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c08b5b5461a73e03"}
19:15:16 [19:15:16.476]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.477]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4db192573340dc3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4db192573340dc3"}
19:15:16 [19:15:16.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78fb2c70a4843f54 epochNumber=1
{"provingJobId":"1:BASE_PARITY:78fb2c70a4843f54"}
19:15:16 [19:15:16.480]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.480]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c18eeae02b10197 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7c18eeae02b10197"}
19:15:16 [19:15:16.482]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7c18eeae02b10197 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7c18eeae02b10197"}
19:15:16 [19:15:16.484]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.485]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3435956376a64b2b epochNumber=1
{"provingJobId":"1:BASE_PARITY:3435956376a64b2b"}
19:15:16 [19:15:16.486]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3435956376a64b2b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3435956376a64b2b"}
19:15:16 [19:15:16.486]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3435956376a64b2b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3435956376a64b2b"}
19:15:16 [19:15:16.487]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3435956376a64b2b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3435956376a64b2b"}
19:15:16 [19:15:16.488]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.489]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a828a8f8161ec398 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a828a8f8161ec398"}
19:15:16 [19:15:16.510]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.511]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fbb59d9e24cf3b5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:fbb59d9e24cf3b5d"}
19:15:16 [19:15:16.512]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:fbb59d9e24cf3b5d
{"provingJobId":"1:BASE_PARITY:fbb59d9e24cf3b5d"}
19:15:16 [19:15:16.513]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fbb59d9e24cf3b5d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fbb59d9e24cf3b5d"}
19:15:16 [19:15:16.513]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fbb59d9e24cf3b5d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:fbb59d9e24cf3b5d"}
19:15:16 [19:15:16.514]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.515]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c40859a4acde74d epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c40859a4acde74d"}
19:15:16 [19:15:16.515]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2c40859a4acde74d
{"provingJobId":"1:BASE_PARITY:2c40859a4acde74d"}
19:15:16 [19:15:16.516]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2c40859a4acde74d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2c40859a4acde74d"}
19:15:16 [19:15:16.517]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.518]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e8da5577c9acb5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:2e8da5577c9acb5e"}
19:15:16 [19:15:16.518]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2e8da5577c9acb5e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2e8da5577c9acb5e"}
19:15:16 [19:15:16.518]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2e8da5577c9acb5e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2e8da5577c9acb5e"}
19:15:16 [19:15:16.520]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7599473e1aea4569 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7599473e1aea4569"}
19:15:16 [19:15:16.521]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7599473e1aea4569 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7599473e1aea4569"}
19:15:16 [19:15:16.521]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7599473e1aea4569 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7599473e1aea4569"}
19:15:16 [19:15:16.522]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.522]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53cad4543577e7b0 epochNumber=0
{"provingJobId":"1:BASE_PARITY:53cad4543577e7b0"}
19:15:16 [19:15:16.523]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:53cad4543577e7b0 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:53cad4543577e7b0"}
19:15:16 [19:15:16.523]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79a015e390adbf4d epochNumber=0
{"provingJobId":"1:BASE_PARITY:79a015e390adbf4d"}
19:15:16 [19:15:16.523]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:53cad4543577e7b0 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:53cad4543577e7b0"}
19:15:16 [19:15:16.523]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6ebf4548694202d epochNumber=0
{"provingJobId":"1:BASE_PARITY:b6ebf4548694202d"}
19:15:16 [19:15:16.524]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.525]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.526]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f24c040643048c7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2f24c040643048c7"}
19:15:16 [19:15:16.526]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:454af24a3d7bdb7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:454af24a3d7bdb7d"}
19:15:16 [19:15:16.527]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b06f82363c034f08 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b06f82363c034f08"}
19:15:16 [19:15:16.528]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.529]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7d4dffbe4eac9713 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7d4dffbe4eac9713"}
19:15:16 [19:15:16.529]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ea32c00a65febbb epochNumber=1
{"provingJobId":"1:BASE_PARITY:0ea32c00a65febbb"}
19:15:16 [19:15:16.529]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1974e29886e53582 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1974e29886e53582"}
19:15:16 [19:15:16.530]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:997c47ed3b647557 epochNumber=1
{"provingJobId":"1:BASE_PARITY:997c47ed3b647557"}
19:15:16 [19:15:16.530]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.531]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd8a006935ab3866 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd8a006935ab3866"}
19:15:16 [19:15:16.531]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d05cef3ed3fe48f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d05cef3ed3fe48f0"}
19:15:16 [19:15:16.531]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b71a70fd27959778 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b71a70fd27959778"}
19:15:16 [19:15:16.532]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:db80c8bb1a3fe85c epochNumber=1
{"provingJobId":"1:BASE_PARITY:db80c8bb1a3fe85c"}
19:15:16 [19:15:16.533]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.533]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:65b63892bd8e556e epochNumber=1
{"provingJobId":"1:BASE_PARITY:65b63892bd8e556e"}
19:15:16 [19:15:16.533]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:63088fabb8ec8b90 epochNumber=1
{"provingJobId":"1:BASE_PARITY:63088fabb8ec8b90"}
19:15:16 [19:15:16.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9a672a564ff4a02 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a9a672a564ff4a02"}
19:15:16 [19:15:16.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:893ccd3519a17c95 epochNumber=1
{"provingJobId":"1:BASE_PARITY:893ccd3519a17c95"}
19:15:16 [19:15:16.535]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.535]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd3b25faddff3e11 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cd3b25faddff3e11"}
19:15:16 [19:15:16.536]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd134fb216952a3f epochNumber=1
{"provingJobId":"1:BASE_PARITY:cd134fb216952a3f"}
19:15:16 [19:15:16.536]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cd3b25faddff3e11 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cd3b25faddff3e11"}
19:15:16 [19:15:16.537]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ed65f3de6f3b4b2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ed65f3de6f3b4b2"}
19:15:16 [19:15:16.538]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b0f03a1b91fdb20 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b0f03a1b91fdb20"}
19:15:16 [19:15:16.538]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6ed65f3de6f3b4b2 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:6ed65f3de6f3b4b2"}
19:15:16 [19:15:16.539]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.539]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10d1aac018fa138a epochNumber=1
{"provingJobId":"1:BASE_PARITY:10d1aac018fa138a"}
19:15:16 [19:15:16.540]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28ef5460095130c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:28ef5460095130c9"}
19:15:16 [19:15:16.540]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:10d1aac018fa138a type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:10d1aac018fa138a"}
19:15:16 [19:15:16.541]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a9f5bcd0e75d5ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:6a9f5bcd0e75d5ad"}
19:15:16 [19:15:16.542]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6a9f5bcd0e75d5ad
{"provingJobId":"1:BASE_PARITY:6a9f5bcd0e75d5ad"}
19:15:16 [19:15:16.542]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6a9f5bcd0e75d5ad type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:6a9f5bcd0e75d5ad"}
19:15:16 [19:15:16.542]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a96398368b0489a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a96398368b0489a1"}
19:15:16 [19:15:16.543]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6a9f5bcd0e75d5ad has already been completed
{"provingJobId":"1:BASE_PARITY:6a9f5bcd0e75d5ad"}
19:15:16 [19:15:16.543]
INFO:
prover-client:proving-broker Proving Broker started
19:15:16 [19:15:16.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b845a62b738f259a epochNumber=1
{"provingJobId":"1:BASE_PARITY:b845a62b738f259a"}
19:15:16 [19:15:16.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca595bee22c5d3c1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ca595bee22c5d3c1"}
19:15:17 [19:15:17.595]
INFO:
prover-client:proving-broker Proving Broker started
19:15:17 [19:15:17.595]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b845a62b738f259a settled=false
{"provingJobId":"1:BASE_PARITY:b845a62b738f259a","status":"pending"}
19:15:17 [19:15:17.596]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ca595bee22c5d3c1 settled=false
{"provingJobId":"1:BASE_PARITY:ca595bee22c5d3c1","status":"pending"}
19:15:17 [19:15:17.596]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b845a62b738f259a type=BASE_PARITY startedAt=1750101316544 older agent has taken job
{"provingJobId":"1:BASE_PARITY:b845a62b738f259a"}
19:15:17 [19:15:17.597]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b845a62b738f259a type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:b845a62b738f259a"}
19:15:17 [19:15:17.598]
INFO:
prover-client:proving-broker Proving Broker started
19:15:17 [19:15:17.598]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb1d395217917c42 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb1d395217917c42"}
19:15:17 [19:15:17.598]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e8adacc53069151 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9e8adacc53069151"}
19:15:18 [19:15:18.598]
INFO:
prover-client:proving-broker Proving Broker started
19:15:18 [19:15:18.599]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:eb1d395217917c42 settled=false
{"provingJobId":"1:BASE_PARITY:eb1d395217917c42","status":"pending"}
19:15:18 [19:15:18.599]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9e8adacc53069151 settled=false
{"provingJobId":"1:BASE_PARITY:9e8adacc53069151","status":"pending"}
19:15:18 [19:15:18.599]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb1d395217917c42 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:eb1d395217917c42"}
19:15:18 [19:15:18.600]
INFO:
prover-client:proving-broker Proving Broker started
19:15:18 [19:15:18.601]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ddfdd9f6d6d6e66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3ddfdd9f6d6d6e66"}
19:15:18 [19:15:18.601]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba4644eb004587bf epochNumber=2
{"provingJobId":"1:BASE_PARITY:ba4644eb004587bf"}
19:15:19 [19:15:19.601]
INFO:
prover-client:proving-broker Proving Broker started
19:15:19 [19:15:19.602]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3ddfdd9f6d6d6e66 settled=false
{"provingJobId":"1:BASE_PARITY:3ddfdd9f6d6d6e66","status":"pending"}
19:15:19 [19:15:19.602]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ba4644eb004587bf settled=false
{"provingJobId":"1:BASE_PARITY:ba4644eb004587bf","status":"pending"}
19:15:19 [19:15:19.602]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3ddfdd9f6d6d6e66 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3ddfdd9f6d6d6e66"}
19:15:19 [19:15:19.603]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3ddfdd9f6d6d6e66 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3ddfdd9f6d6d6e66"}
19:15:19 [19:15:19.604]
INFO:
prover-client:proving-broker Proving Broker started
19:15:19 [19:15:19.605]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ebf4d74bc28298f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ebf4d74bc28298f2"}
19:15:19 [19:15:19.605]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5fdaaf0edae75990 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5fdaaf0edae75990"}
19:15:19 [19:15:19.605]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ebf4d74bc28298f2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ebf4d74bc28298f2"}
19:15:19 [19:15:19.606]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5fdaaf0edae75990 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5fdaaf0edae75990"}
19:15:19 [19:15:19.607]
INFO:
prover-client:proving-broker Proving Broker started
19:15:19 [19:15:19.607]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5e8674985ac4932 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5e8674985ac4932"}
19:15:19 [19:15:19.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:661370e4a2c4aab3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:661370e4a2c4aab3"}
19:15:19 [19:15:19.608]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e5e8674985ac4932 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e5e8674985ac4932"}
19:15:19 [19:15:19.608]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e5e8674985ac4932 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e5e8674985ac4932"}
19:15:19 [19:15:19.608]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:661370e4a2c4aab3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:661370e4a2c4aab3"}
19:15:19 [19:15:19.609]
INFO:
prover-client:proving-broker Proving Broker started
19:15:19 [19:15:19.610]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bc46604b59d662b8 err=test error
{"provingJoId":"1:BASE_PARITY:bc46604b59d662b8"}
19:15:19 [19:15:19.611]
INFO:
prover-client:proving-broker Proving Broker started
19:15:19 [19:15:19.611]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4dca3a72b3abb585 not found
{"provingJobId":"1:BASE_PARITY:4dca3a72b3abb585"}
19:15:19 [19:15:19.612]
INFO:
prover-client:proving-broker Proving Broker started
19:15:19 [19:15:19.613]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc42dee5b48cfc09 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc42dee5b48cfc09"}
19:15:19 [19:15:19.614]
INFO:
prover-client:proving-broker Proving Broker started
19:15:19 [19:15:19.614]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ff19699da4f2d1c epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ff19699da4f2d1c"}
19:15:19 [19:15:19.714]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7ff19699da4f2d1c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:7ff19699da4f2d1c"}
19:15:19 [19:15:19.715]
INFO:
prover-client:proving-broker Proving Broker started
19:15:19 [19:15:19.716]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d26cb3520f7682b epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d26cb3520f7682b"}
19:15:19 [19:15:19.816]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6d26cb3520f7682b timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6d26cb3520f7682b"}
19:15:19 [19:15:19.823]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1380c9f54d138ba4 epochNumber=10
{"provingJobId":"1:BASE_PARITY:1380c9f54d138ba4"}
19:15:19 [19:15:19.842]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:19 [19:15:19.949]
INFO:
prover-client:proving-broker Proving Broker started
19:15:19 [19:15:19.950]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f701cdec8be7aa69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f701cdec8be7aa69"}
19:15:20 [19:15:20.102]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f701cdec8be7aa69 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f701cdec8be7aa69"}
19:15:20 [19:15:20.153]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.153]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:651ed626b11ee593 epochNumber=1
{"provingJobId":"1:BASE_PARITY:651ed626b11ee593"}
19:15:20 [19:15:20.154]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:651ed626b11ee593 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:651ed626b11ee593"}
19:15:20 [19:15:20.155]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.155]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2bb2e2eefbb4f31a epochNumber=1
{"provingJobId":"1:BASE_PARITY:2bb2e2eefbb4f31a"}
19:15:20 [19:15:20.156]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2bb2e2eefbb4f31a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:2bb2e2eefbb4f31a"}
19:15:20 [19:15:20.156]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2bb2e2eefbb4f31a type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:2bb2e2eefbb4f31a"}
19:15:20 [19:15:20.157]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.158]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:618dc7e74fd140cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:618dc7e74fd140cd"}
19:15:20 [19:15:20.158]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:618dc7e74fd140cd type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:618dc7e74fd140cd"}
19:15:20 [19:15:20.159]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82a6b58b1b60f3bf epochNumber=1
{"provingJobId":"1:BASE_PARITY:82a6b58b1b60f3bf"}
19:15:20 [19:15:20.160]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:82a6b58b1b60f3bf type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:82a6b58b1b60f3bf"}
19:15:20 [19:15:20.160]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e67ec4c01bfc394c epochNumber=3
{"provingJobId":"1:BASE_PARITY:e67ec4c01bfc394c"}
19:15:20 [19:15:20.184]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:20 [19:15:20.186]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:82a6b58b1b60f3bf err=test error
{"provingJoId":"1:BASE_PARITY:82a6b58b1b60f3bf"}
19:15:20 [19:15:20.187]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.187]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:89b6dc90c87f0173 settled=false
{"provingJobId":"1:BASE_PARITY:89b6dc90c87f0173","status":"pending"}
19:15:20 [19:15:20.188]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2004845575c57d8e settled=false
{"provingJobId":"1:BASE_PARITY:2004845575c57d8e","status":"pending"}
19:15:20 [19:15:20.190]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.190]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a0c54bf787b0efe4 settled=true
{"provingJobId":"1:BASE_PARITY:a0c54bf787b0efe4","status":"fulfilled"}
19:15:20 [19:15:20.190]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:cb8711769dd63e7d settled=true
{"provingJobId":"2:BASE_PARITY:cb8711769dd63e7d","status":"fulfilled"}
19:15:20 [19:15:20.191]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.191]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8456c8986e5fecb0 settled=true
{"provingJobId":"1:BASE_PARITY:8456c8986e5fecb0","status":"fulfilled"}
19:15:20 [19:15:20.192]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c3fa28dd0fe4e268 settled=false
{"provingJobId":"1:BASE_PARITY:c3fa28dd0fe4e268","status":"pending"}
19:15:20 [19:15:20.193]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.194]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:169690bb1f55ca58 epochNumber=1
{"provingJobId":"1:BASE_PARITY:169690bb1f55ca58"}
19:15:20 [19:15:20.195]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.195]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1fdaa9a32ec1eb1b epochNumber=1
{"provingJobId":"1:BASE_PARITY:1fdaa9a32ec1eb1b"}
19:15:20 [19:15:20.196]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:1fdaa9a32ec1eb1b: Error: db error: Error: db error
19:15:20 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:1fdaa9a32ec1eb1b"}
19:15:20 [19:15:20.197]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.197]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:859da9cdbb645cdf epochNumber=1
{"provingJobId":"1:BASE_PARITY:859da9cdbb645cdf"}
19:15:20 [19:15:20.198]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:859da9cdbb645cdf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:859da9cdbb645cdf"}
19:15:20 [19:15:20.198]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:859da9cdbb645cdf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:859da9cdbb645cdf"}
19:15:20 [19:15:20.199]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.199]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d9990cf70442267 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d9990cf70442267"}
19:15:20 [19:15:20.200]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8d9990cf70442267 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8d9990cf70442267"}
19:15:20 [19:15:20.200]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8d9990cf70442267 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8d9990cf70442267"}
19:15:20 [19:15:20.200]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:8d9990cf70442267: Error: db error
19:15:20 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:8d9990cf70442267"}
19:15:20 [19:15:20.201]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.202]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b791f5fcf075dcf4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b791f5fcf075dcf4"}
19:15:20 [19:15:20.202]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b791f5fcf075dcf4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b791f5fcf075dcf4"}
19:15:20 [19:15:20.202]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b791f5fcf075dcf4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b791f5fcf075dcf4"}
19:15:20 [19:15:20.204]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.204]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd9084de730b91ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd9084de730b91ad"}
19:15:20 [19:15:20.204]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bd9084de730b91ad type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bd9084de730b91ad"}
19:15:20 [19:15:20.204]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bd9084de730b91ad type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bd9084de730b91ad"}
19:15:20 [19:15:20.205]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:bd9084de730b91ad jobErr=test error: Error: db error
19:15:20 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:bd9084de730b91ad"}
19:15:20 [19:15:20.206]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.206]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ee50a507753a1443 not found
{"provingJobId":"1:BASE_PARITY:ee50a507753a1443"}
19:15:20 [19:15:20.208]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.208]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ad4a3e4264d052a3 err=test error
{"provingJoId":"1:BASE_PARITY:ad4a3e4264d052a3"}
19:15:20 [19:15:20.209]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.219]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2707fd50ac07bc77 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2707fd50ac07bc77"}
19:15:20 [19:15:20.220]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2707fd50ac07bc77 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2707fd50ac07bc77"}
19:15:20 [19:15:20.220]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2707fd50ac07bc77 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2707fd50ac07bc77"}
19:15:20 [19:15:20.231]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:6291107403ce1a38 epochNumber=2
{"provingJobId":"2:BASE_PARITY:6291107403ce1a38"}
19:15:20 [19:15:20.231]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:6291107403ce1a38 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:6291107403ce1a38"}
19:15:20 [19:15:20.231]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:6291107403ce1a38 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:6291107403ce1a38"}
19:15:20 [19:15:20.292]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:ebf3d0a353c3ba4e epochNumber=3
{"provingJobId":"3:BASE_PARITY:ebf3d0a353c3ba4e"}
19:15:20 [19:15:20.310]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:20 [19:15:20.353]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:f17a1ab3ec6d50d4 epochNumber=4
{"provingJobId":"4:BASE_PARITY:f17a1ab3ec6d50d4"}
19:15:20 [19:15:20.361]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:20 [19:15:20.415]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:7ab61b056a8ab180 epochNumber=5
{"provingJobId":"5:BASE_PARITY:7ab61b056a8ab180"}
19:15:20 [19:15:20.436]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:20 [19:15:20.468]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.468]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d807c0c3734776bc epochNumber=42
{"provingJobId":"1:BASE_PARITY:d807c0c3734776bc"}
19:15:20 [19:15:20.469]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testZuQZ2K/42 with map size 1073741824
19:15:20 [19:15:20.472]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.493]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:5f0de3bdd4f1d0a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f0de3bdd4f1d0a7"}
19:15:20 [19:15:20.496]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.497]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b3a95c5275774f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b3a95c5275774f8"}
19:15:20 [19:15:20.497]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiDhlMF/1 with map size 1073741824
19:15:20 [19:15:20.499]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.516]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d119ab363a202347 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d119ab363a202347"}
19:15:20 [19:15:20.524]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.525]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:326e24a3b1b6f264 epochNumber=1
{"provingJobId":"1:BASE_PARITY:326e24a3b1b6f264"}
19:15:20 [19:15:20.525]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfNMHPF/1 with map size 1073741824
19:15:20 [19:15:20.526]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.549]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:326e24a3b1b6f264 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:326e24a3b1b6f264"}
19:15:20 [19:15:20.552]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.553]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d77bcfa74ae7acb1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d77bcfa74ae7acb1"}
19:15:20 [19:15:20.555]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoq0mpl/1 with map size 1073741824
19:15:20 [19:15:20.556]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.577]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d77bcfa74ae7acb1 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d77bcfa74ae7acb1"}
19:15:20 [19:15:20.578]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d77bcfa74ae7acb1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d77bcfa74ae7acb1"}
19:15:20 [19:15:20.584]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d77bcfa74ae7acb1 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d77bcfa74ae7acb1"}
19:15:20 [19:15:20.588]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.588]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40ee9f2ee0b169fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:40ee9f2ee0b169fa"}
19:15:20 [19:15:20.589]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTKP7Ti/1 with map size 1073741824
19:15:20 [19:15:20.593]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.613]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.615]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.616]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8972f0b816b40fef epochNumber=1
{"provingJobId":"1:BASE_PARITY:8972f0b816b40fef"}
19:15:20 [19:15:20.617]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf5CMju/1 with map size 1073741824
19:15:20 [19:15:20.618]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.634]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:8972f0b816b40fef
{"provingJobId":"1:BASE_PARITY:8972f0b816b40fef"}
19:15:20 [19:15:20.635]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8972f0b816b40fef type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8972f0b816b40fef"}
19:15:20 [19:15:20.635]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8972f0b816b40fef type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:8972f0b816b40fef"}
19:15:20 [19:15:20.642]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.643]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f9334284849417f4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f9334284849417f4"}
19:15:20 [19:15:20.644]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testothlaE/1 with map size 1073741824
19:15:20 [19:15:20.645]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.661]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f9334284849417f4
{"provingJobId":"1:BASE_PARITY:f9334284849417f4"}
19:15:20 [19:15:20.661]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f9334284849417f4 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f9334284849417f4"}
19:15:20 [19:15:20.669]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.669]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83013ea520bb478b epochNumber=1
{"provingJobId":"1:BASE_PARITY:83013ea520bb478b"}
19:15:20 [19:15:20.670]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxTH2Nw/1 with map size 1073741824
19:15:20 [19:15:20.671]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.687]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:83013ea520bb478b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:83013ea520bb478b"}
19:15:20 [19:15:20.688]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:83013ea520bb478b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:83013ea520bb478b"}
19:15:20 [19:15:20.696]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.696]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d530463294916514 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d530463294916514"}
19:15:20 [19:15:20.698]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testESgDaw/1 with map size 1073741824
19:15:20 [19:15:20.702]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.717]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d530463294916514 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d530463294916514"}
19:15:20 [19:15:20.717]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d530463294916514 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d530463294916514"}
19:15:20 [19:15:20.724]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.724]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57b0cff3b053ee40 epochNumber=0
{"provingJobId":"1:BASE_PARITY:57b0cff3b053ee40"}
19:15:20 [19:15:20.725]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testQsSzqA/0 with map size 1073741824
19:15:20 [19:15:20.726]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.743]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:57b0cff3b053ee40 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:57b0cff3b053ee40"}
19:15:20 [19:15:20.743]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8fb493d9d5ddcdd epochNumber=0
{"provingJobId":"1:BASE_PARITY:e8fb493d9d5ddcdd"}
19:15:20 [19:15:20.744]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:57b0cff3b053ee40 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:57b0cff3b053ee40"}
19:15:20 [19:15:20.744]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c03347b615b904a epochNumber=0
{"provingJobId":"1:BASE_PARITY:5c03347b615b904a"}
19:15:20 [19:15:20.757]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.759]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.760]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:429e8e1e3d47c0f3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:429e8e1e3d47c0f3"}
19:15:20 [19:15:20.760]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIAvR4w/2 with map size 1073741824
19:15:20 [19:15:20.761]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fbb233d829363d14 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fbb233d829363d14"}
19:15:20 [19:15:20.776]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIAvR4w/1 with map size 1073741824
19:15:20 [19:15:20.777]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.796]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.796]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34208512a24ac2a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:34208512a24ac2a0"}
19:15:20 [19:15:20.797]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMpuEes/1 with map size 1073741824
19:15:20 [19:15:20.799]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.818]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.819]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7b32f568721f95b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7b32f568721f95b"}
19:15:20 [19:15:20.821]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH2RoB4/1 with map size 1073741824
19:15:20 [19:15:20.822]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.839]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2af45b9b8919b1e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2af45b9b8919b1e9"}
19:15:20 [19:15:20.847]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4ab0652579ec690 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f4ab0652579ec690"}
19:15:20 [19:15:20.848]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testH2RoB4/2 with map size 1073741824
19:15:20 [19:15:20.850]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.866]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c24cd17aa2b27e5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c24cd17aa2b27e5e"}
19:15:20 [19:15:20.873]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.874]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af068a3b8bf1fa3b epochNumber=1
{"provingJobId":"1:BASE_PARITY:af068a3b8bf1fa3b"}
19:15:20 [19:15:20.874]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaCalps/1 with map size 1073741824
19:15:20 [19:15:20.876]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.893]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b98a62a822180046 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b98a62a822180046"}
19:15:20 [19:15:20.897]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f99f90ffb9e4f192 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f99f90ffb9e4f192"}
19:15:20 [19:15:20.898]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testaCalps/2 with map size 1073741824
19:15:20 [19:15:20.900]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.916]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a518cf70f882399 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a518cf70f882399"}
19:15:20 [19:15:20.924]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.925]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08c7bcb0e2d7dbf0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:08c7bcb0e2d7dbf0"}
19:15:20 [19:15:20.927]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnDZjtf/1 with map size 1073741824
19:15:20 [19:15:20.928]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c67c8f1643c7842 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c67c8f1643c7842"}
19:15:20 [19:15:20.951]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58373b723982de23 epochNumber=2
{"provingJobId":"1:BASE_PARITY:58373b723982de23"}
19:15:20 [19:15:20.953]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnDZjtf/2 with map size 1073741824
19:15:20 [19:15:20.955]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.972]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d0680048b4a08f47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d0680048b4a08f47"}
19:15:20 [19:15:20.980]
INFO:
prover-client:proving-broker Proving Broker started
19:15:20 [19:15:20.980]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7bac1f3a04f7e0a epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7bac1f3a04f7e0a"}
19:15:20 [19:15:20.981]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNUifZG/1 with map size 1073741824
19:15:20 [19:15:20.982]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:20 [19:15:20.998]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:369cfbff98f603a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:369cfbff98f603a0"}
19:15:21 [19:15:21.003]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b7bac1f3a04f7e0a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b7bac1f3a04f7e0a"}
19:15:21 [19:15:21.011]
INFO:
prover-client:proving-broker Proving Broker started
19:15:21 [19:15:21.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:504fd1a43d031c58 epochNumber=1
{"provingJobId":"1:BASE_PARITY:504fd1a43d031c58"}
19:15:21 [19:15:21.014]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7KIIZd/1 with map size 1073741824
19:15:21 [19:15:21.016]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:21 [19:15:21.035]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:755cfc2fcf848602 epochNumber=1
{"provingJobId":"1:BASE_PARITY:755cfc2fcf848602"}
19:15:21 [19:15:21.039]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:504fd1a43d031c58 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:504fd1a43d031c58"}
19:15:21 [19:15:21.049]
INFO:
prover-client:proving-broker Proving Broker started
19:15:21 [19:15:21.050]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b4b4cc74e8e2af3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b4b4cc74e8e2af3"}
19:15:21 [19:15:21.051]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAFoyGG/1 with map size 1073741824
19:15:21 [19:15:21.053]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:21 [19:15:21.068]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:665755b140e21671 epochNumber=1
{"provingJobId":"1:BASE_PARITY:665755b140e21671"}
19:15:21 [19:15:21.073]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3b4b4cc74e8e2af3 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:3b4b4cc74e8e2af3"}
19:15:21 [19:15:21.078]
INFO:
prover-client:proving-broker Proving Broker started
19:15:21 [19:15:21.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5473e620fa237bcf epochNumber=1
{"provingJobId":"1:BASE_PARITY:5473e620fa237bcf"}
19:15:21 [19:15:21.080]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqjcvB1/1 with map size 1073741824
19:15:21 [19:15:21.082]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:21 [19:15:21.099]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5473e620fa237bcf
{"provingJobId":"1:BASE_PARITY:5473e620fa237bcf"}
19:15:21 [19:15:21.099]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5473e620fa237bcf type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5473e620fa237bcf"}
19:15:21 [19:15:21.105]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3eaaa2fa2f54eb4c epochNumber=1
{"provingJobId":"1:BASE_PARITY:3eaaa2fa2f54eb4c"}
19:15:21 [19:15:21.112]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5473e620fa237bcf has already been completed
{"provingJobId":"1:BASE_PARITY:5473e620fa237bcf"}
19:15:21 [19:15:21.114]
INFO:
prover-client:proving-broker Proving Broker started
19:15:21 [19:15:21.115]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:127aa07fb5885496 epochNumber=1
{"provingJobId":"1:BASE_PARITY:127aa07fb5885496"}
19:15:21 [19:15:21.116]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8QFtdq/1 with map size 1073741824
19:15:21 [19:15:21.117]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:21 [19:15:21.132]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d9ef1cdf0e4eb085 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d9ef1cdf0e4eb085"}
19:15:21 [19:15:21.133]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8QFtdq/2 with map size 1073741824
19:15:21 [19:15:21.134]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:22 [19:15:22.200]
INFO:
prover-client:proving-broker Proving Broker started
19:15:22 [19:15:22.204]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:127aa07fb5885496 settled=false
{"provingJobId":"1:BASE_PARITY:127aa07fb5885496","status":"pending"}
19:15:22 [19:15:22.206]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d9ef1cdf0e4eb085 settled=false
{"provingJobId":"1:BASE_PARITY:d9ef1cdf0e4eb085","status":"pending"}
19:15:22 [19:15:22.207]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:127aa07fb5885496 type=BASE_PARITY startedAt=1750101321149 older agent has taken job
{"provingJobId":"1:BASE_PARITY:127aa07fb5885496"}
19:15:22 [19:15:22.207]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:127aa07fb5885496 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:127aa07fb5885496"}
19:15:22 [19:15:22.210]
INFO:
prover-client:proving-broker Proving Broker started
19:15:22 [19:15:22.210]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:492d329fa4f39f38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:492d329fa4f39f38"}
19:15:22 [19:15:22.211]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testymDvlx/1 with map size 1073741824
19:15:22 [19:15:22.213]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:22 [19:15:22.228]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0099a71cc4186a87 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0099a71cc4186a87"}
19:15:22 [19:15:22.228]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testymDvlx/2 with map size 1073741824
19:15:22 [19:15:22.230]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:23 [19:15:23.246]
INFO:
prover-client:proving-broker Proving Broker started
19:15:23 [19:15:23.247]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:492d329fa4f39f38 settled=false
{"provingJobId":"1:BASE_PARITY:492d329fa4f39f38","status":"pending"}
19:15:23 [19:15:23.249]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0099a71cc4186a87 settled=false
{"provingJobId":"1:BASE_PARITY:0099a71cc4186a87","status":"pending"}
19:15:23 [19:15:23.249]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:492d329fa4f39f38 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:492d329fa4f39f38"}
19:15:23 [19:15:23.252]
INFO:
prover-client:proving-broker Proving Broker started
19:15:23 [19:15:23.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7e43b69dd7cc5e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7e43b69dd7cc5e0"}
19:15:23 [19:15:23.253]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7nsVYC/1 with map size 1073741824
19:15:23 [19:15:23.255]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:23 [19:15:23.271]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e436218a52a51fba epochNumber=2
{"provingJobId":"1:BASE_PARITY:e436218a52a51fba"}
19:15:23 [19:15:23.272]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7nsVYC/2 with map size 1073741824
19:15:23 [19:15:23.273]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:24 [19:15:24.288]
INFO:
prover-client:proving-broker Proving Broker started
19:15:24 [19:15:24.291]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f7e43b69dd7cc5e0 settled=false
{"provingJobId":"1:BASE_PARITY:f7e43b69dd7cc5e0","status":"pending"}
19:15:24 [19:15:24.292]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e436218a52a51fba settled=false
{"provingJobId":"1:BASE_PARITY:e436218a52a51fba","status":"pending"}
19:15:24 [19:15:24.294]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f7e43b69dd7cc5e0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f7e43b69dd7cc5e0"}
19:15:24 [19:15:24.294]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f7e43b69dd7cc5e0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f7e43b69dd7cc5e0"}
19:15:24 [19:15:24.306]
INFO:
prover-client:proving-broker Proving Broker started
19:15:24 [19:15:24.307]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2c0b7ad24021578 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2c0b7ad24021578"}
19:15:24 [19:15:24.307]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIuiXBy/1 with map size 1073741824
19:15:24 [19:15:24.309]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:24 [19:15:24.324]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd1de18c85fc15a1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:bd1de18c85fc15a1"}
19:15:24 [19:15:24.325]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIuiXBy/2 with map size 1073741824
19:15:24 [19:15:24.326]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:24 [19:15:24.342]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f2c0b7ad24021578 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f2c0b7ad24021578"}
19:15:24 [19:15:24.349]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bd1de18c85fc15a1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bd1de18c85fc15a1"}
19:15:24 [19:15:24.358]
INFO:
prover-client:proving-broker Proving Broker started
19:15:24 [19:15:24.359]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8a6d2fe1f7d5758 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f8a6d2fe1f7d5758"}
19:15:24 [19:15:24.359]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnIBK37/1 with map size 1073741824
19:15:24 [19:15:24.360]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:24 [19:15:24.376]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40b47f3431d9fec8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:40b47f3431d9fec8"}
19:15:24 [19:15:24.377]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnIBK37/2 with map size 1073741824
19:15:24 [19:15:24.378]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:24 [19:15:24.394]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f8a6d2fe1f7d5758 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f8a6d2fe1f7d5758"}
19:15:24 [19:15:24.394]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f8a6d2fe1f7d5758 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f8a6d2fe1f7d5758"}
19:15:24 [19:15:24.400]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:40b47f3431d9fec8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:40b47f3431d9fec8"}
19:15:24 [19:15:24.407]
INFO:
prover-client:proving-broker Proving Broker started
19:15:24 [19:15:24.408]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0cf200314da65ef8 err=test error
{"provingJoId":"1:BASE_PARITY:0cf200314da65ef8"}
19:15:24 [19:15:24.410]
INFO:
prover-client:proving-broker Proving Broker started
19:15:24 [19:15:24.411]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:35571aac9220dad4 not found
{"provingJobId":"1:BASE_PARITY:35571aac9220dad4"}
19:15:24 [19:15:24.415]
INFO:
prover-client:proving-broker Proving Broker started
19:15:24 [19:15:24.415]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28caad127217d137 epochNumber=1
{"provingJobId":"1:BASE_PARITY:28caad127217d137"}
19:15:24 [19:15:24.416]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testujq5QB/1 with map size 1073741824
19:15:24 [19:15:24.417]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:24 [19:15:24.434]
INFO:
prover-client:proving-broker Proving Broker started
19:15:24 [19:15:24.435]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f56abf62d9bfd782 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f56abf62d9bfd782"}
19:15:24 [19:15:24.436]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8oath8/1 with map size 1073741824
19:15:24 [19:15:24.438]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:24 [19:15:24.561]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f56abf62d9bfd782 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f56abf62d9bfd782"}
19:15:24 [19:15:24.568]
INFO:
prover-client:proving-broker Proving Broker started
19:15:24 [19:15:24.569]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de78bba27bfe1e81 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de78bba27bfe1e81"}
19:15:24 [19:15:24.569]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkzii3y/1 with map size 1073741824
19:15:24 [19:15:24.571]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:24 [19:15:24.696]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:de78bba27bfe1e81 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:de78bba27bfe1e81"}
19:15:24 [19:15:24.699]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc8f7f42c905cdbe epochNumber=10
{"provingJobId":"1:BASE_PARITY:dc8f7f42c905cdbe"}
19:15:24 [19:15:24.700]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testkzii3y/10 with map size 1073741824
19:15:24 [19:15:24.701]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:24 [19:15:24.721]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:24 [19:15:24.722]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:15:24 [19:15:24.843]
INFO:
prover-client:proving-broker Proving Broker started
19:15:24 [19:15:24.844]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14860ea95ccda967 epochNumber=1
{"provingJobId":"1:BASE_PARITY:14860ea95ccda967"}
19:15:24 [19:15:24.845]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUsB1yr/1 with map size 1073741824
19:15:24 [19:15:24.847]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.023]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:14860ea95ccda967 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:14860ea95ccda967"}
19:15:25 [19:15:25.065]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.066]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8df5822b564fcaf6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8df5822b564fcaf6"}
19:15:25 [19:15:25.066]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVw2DyO/1 with map size 1073741824
19:15:25 [19:15:25.067]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.083]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8df5822b564fcaf6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:8df5822b564fcaf6"}
19:15:25 [19:15:25.086]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf24119693dcc1b7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf24119693dcc1b7"}
19:15:25 [19:15:25.087]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQRbVrH/1 with map size 1073741824
19:15:25 [19:15:25.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.103]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cf24119693dcc1b7 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:cf24119693dcc1b7"}
19:15:25 [19:15:25.104]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cf24119693dcc1b7 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:cf24119693dcc1b7"}
19:15:25 [19:15:25.112]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.113]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e3be77a11350b1a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e3be77a11350b1a"}
19:15:25 [19:15:25.113]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFSwc9s/1 with map size 1073741824
19:15:25 [19:15:25.114]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.128]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6e3be77a11350b1a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6e3be77a11350b1a"}
19:15:25 [19:15:25.136]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.136]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed224917fda1cf9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed224917fda1cf9f"}
19:15:25 [19:15:25.138]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLbGsL4/1 with map size 1073741824
19:15:25 [19:15:25.139]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.154]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ed224917fda1cf9f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:ed224917fda1cf9f"}
19:15:25 [19:15:25.154]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26018bd132283d46 epochNumber=3
{"provingJobId":"1:BASE_PARITY:26018bd132283d46"}
19:15:25 [19:15:25.155]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testLbGsL4/3 with map size 1073741824
19:15:25 [19:15:25.156]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.186]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:25 [19:15:25.187]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:15:25 [19:15:25.195]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ed224917fda1cf9f err=test error
{"provingJoId":"1:BASE_PARITY:ed224917fda1cf9f"}
19:15:25 [19:15:25.198]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZhDRqW/1 with map size 1073741824
19:15:25 [19:15:25.199]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.214]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZhDRqW/2 with map size 1073741824
19:15:25 [19:15:25.216]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.230]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.231]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:32d6d3404cfbb4c3 settled=false
{"provingJobId":"1:BASE_PARITY:32d6d3404cfbb4c3","status":"pending"}
19:15:25 [19:15:25.232]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f03d57f36979c887 settled=false
{"provingJobId":"1:BASE_PARITY:f03d57f36979c887","status":"pending"}
19:15:25 [19:15:25.238]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2M2Tl5/1 with map size 1073741824
19:15:25 [19:15:25.239]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.255]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2M2Tl5/2 with map size 1073741824
19:15:25 [19:15:25.256]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.280]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.282]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5c58a83a6b294468 settled=true
{"provingJobId":"1:BASE_PARITY:5c58a83a6b294468","status":"fulfilled"}
19:15:25 [19:15:25.283]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:3fbd05e30c8d3ee2 settled=true
{"provingJobId":"2:BASE_PARITY:3fbd05e30c8d3ee2","status":"fulfilled"}
19:15:25 [19:15:25.287]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPIH7XC/1 with map size 1073741824
19:15:25 [19:15:25.289]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.308]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testPIH7XC/2 with map size 1073741824
19:15:25 [19:15:25.309]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.323]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.325]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3f455faac3c42559 settled=true
{"provingJobId":"1:BASE_PARITY:3f455faac3c42559","status":"fulfilled"}
19:15:25 [19:15:25.327]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:232abb89aeb600ab settled=false
{"provingJobId":"1:BASE_PARITY:232abb89aeb600ab","status":"pending"}
19:15:25 [19:15:25.331]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3ae5446e5593720 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3ae5446e5593720"}
19:15:25 [19:15:25.332]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9ocp0J/1 with map size 1073741824
19:15:25 [19:15:25.333]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.350]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.350]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f45309e537e9fd9c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f45309e537e9fd9c"}
19:15:25 [19:15:25.351]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:f45309e537e9fd9c: Error: db error: Error: db error
19:15:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:f45309e537e9fd9c"}
19:15:25 [19:15:25.354]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.354]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e96b20c551d6df42 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e96b20c551d6df42"}
19:15:25 [19:15:25.355]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCvfZn5/1 with map size 1073741824
19:15:25 [19:15:25.356]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.370]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e96b20c551d6df42 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e96b20c551d6df42"}
19:15:25 [19:15:25.371]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e96b20c551d6df42 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e96b20c551d6df42"}
19:15:25 [19:15:25.379]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.379]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e46f393d8f30bc5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e46f393d8f30bc5"}
19:15:25 [19:15:25.380]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGemd2o/1 with map size 1073741824
19:15:25 [19:15:25.381]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.395]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1e46f393d8f30bc5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1e46f393d8f30bc5"}
19:15:25 [19:15:25.395]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1e46f393d8f30bc5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1e46f393d8f30bc5"}
19:15:25 [19:15:25.396]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:1e46f393d8f30bc5: Error: db error
19:15:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:1e46f393d8f30bc5"}
19:15:25 [19:15:25.399]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.400]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f95f9330544060a epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f95f9330544060a"}
19:15:25 [19:15:25.401]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH2U0u5/1 with map size 1073741824
19:15:25 [19:15:25.402]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.416]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5f95f9330544060a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5f95f9330544060a"}
19:15:25 [19:15:25.417]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5f95f9330544060a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5f95f9330544060a"}
19:15:25 [19:15:25.425]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.426]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c8ba5b08818cbc43 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c8ba5b08818cbc43"}
19:15:25 [19:15:25.426]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUJyDHA/1 with map size 1073741824
19:15:25 [19:15:25.428]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.442]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c8ba5b08818cbc43 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c8ba5b08818cbc43"}
19:15:25 [19:15:25.442]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c8ba5b08818cbc43 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c8ba5b08818cbc43"}
19:15:25 [19:15:25.443]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:c8ba5b08818cbc43 jobErr=test error: Error: db error
19:15:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:c8ba5b08818cbc43"}
19:15:25 [19:15:25.447]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.447]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4b49844b6b9a4e73 not found
{"provingJobId":"1:BASE_PARITY:4b49844b6b9a4e73"}
19:15:25 [19:15:25.450]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.451]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0824acbfa5ef3299 err=test error
{"provingJoId":"1:BASE_PARITY:0824acbfa5ef3299"}
19:15:25 [19:15:25.454]
INFO:
prover-client:proving-broker Proving Broker started
19:15:25 [19:15:25.465]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd02887d7214fd71 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd02887d7214fd71"}
19:15:25 [19:15:25.466]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testp8tzMt/1 with map size 1073741824
19:15:25 [19:15:25.467]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.481]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bd02887d7214fd71 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bd02887d7214fd71"}
19:15:25 [19:15:25.482]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bd02887d7214fd71 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bd02887d7214fd71"}
19:15:25 [19:15:25.497]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:3de7c5160e99d493 epochNumber=2
{"provingJobId":"2:BASE_PARITY:3de7c5160e99d493"}
19:15:25 [19:15:25.497]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testp8tzMt/2 with map size 1073741824
19:15:25 [19:15:25.499]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.513]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:3de7c5160e99d493 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:3de7c5160e99d493"}
19:15:25 [19:15:25.513]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:3de7c5160e99d493 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:3de7c5160e99d493"}
19:15:25 [19:15:25.578]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:5f19ac718d29fb11 epochNumber=3
{"provingJobId":"3:BASE_PARITY:5f19ac718d29fb11"}
19:15:25 [19:15:25.579]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testp8tzMt/3 with map size 1073741824
19:15:25 [19:15:25.580]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.605]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:25 [19:15:25.605]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:15:25 [19:15:25.656]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:6b74cd7e93a3f0f8 epochNumber=4
{"provingJobId":"4:BASE_PARITY:6b74cd7e93a3f0f8"}
19:15:25 [19:15:25.657]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testp8tzMt/4 with map size 1073741824
19:15:25 [19:15:25.658]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.682]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:25 [19:15:25.682]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
19:15:25 [19:15:25.735]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:602d816f0c11643a epochNumber=5
{"provingJobId":"5:BASE_PARITY:602d816f0c11643a"}
19:15:25 [19:15:25.736]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testp8tzMt/5 with map size 1073741824
19:15:25 [19:15:25.737]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:15:25 [19:15:25.761]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:15:25 [19:15:25.761]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
19:15:25
PASS src/proving_broker/proving_broker.test.ts (
11.26 s)
19:15:25 ProvingBroker
19:15:25 Producer API
19:15:25
✓ refuses stale jobs (34 ms)
19:15:25
✓ enqueues jobs (4 ms)
19:15:25
✓ ignores duplicate jobs (4 ms)
19:15:25
✓ reports correct status when enqueuing repeat jobs (4 ms)
19:15:25
✓ throws an error in case of duplicate job IDs (21 ms)
19:15:25
✓ returns not-found status for non-existing jobs (1 ms)
19:15:25
✓ cancels jobs in queue (3 ms)
19:15:25
✓ cancels jobs in-progress (2 ms)
19:15:25
✓ returns job result if successful (3 ms)
19:15:25
✓ returns job error if failed (2 ms)
19:15:25
✓ correctly returns job status for concurrent writes (2 ms)
19:15:25
✓ refuses stale jobs (29 ms)
19:15:25
✓ enqueues jobs (27 ms)
19:15:25
✓ ignores duplicate jobs (28 ms)
19:15:25
✓ reports correct status when enqueuing repeat jobs (35 ms)
19:15:25
✓ throws an error in case of duplicate job IDs (27 ms)
19:15:25
✓ returns not-found status for non-existing jobs (2 ms)
19:15:25
✓ cancels jobs in queue (27 ms)
19:15:25
✓ cancels jobs in-progress (25 ms)
19:15:25
✓ returns job result if successful (27 ms)
19:15:25
✓ returns job error if failed (28 ms)
19:15:25
✓ correctly returns job status for concurrent writes (33 ms)
19:15:25 Consumer API
19:15:25
✓ returns undefined if no jobs are available (1 ms)
19:15:25
✓ returns jobs in priority order (2 ms)
19:15:25
✓ returns undefined if no jobs are available for the given allowList (1 ms)
19:15:25
✓ returns a job if it is in the allowList (2 ms)
19:15:25
✓ returns the most important job if it is in the allowList (2 ms)
19:15:25
✓ returns any job if filter is empty (2 ms)
19:15:25
✓ returns a new job when reporting job success (2 ms)
19:15:25
✓ returns a new job when reporting permanent error (1 ms)
19:15:25
✓ returns a new job when reporting retry-able error (1 ms)
19:15:25
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
19:15:25
✓ returns a new job if job is already in progress elsewhere (1054 ms)
19:15:25
✓ avoids sending the same job to a new agent after a restart (1003 ms)
19:15:25
✓ avoids sending a completed job to a new agent after a restart (1003 ms)
19:15:25
✓ tracks job result if in progress (2 ms)
19:15:25
✓ tracks job result even if job is in queue (2 ms)
19:15:25
✓ ignores reported job error if unknown job (1 ms)
19:15:25
✓ ignores job result if unknown job (1 ms)
19:15:25
✓ returns undefined if no jobs are available (1 ms)
19:15:25
✓ returns jobs in priority order (36 ms)
19:15:25
✓ returns undefined if no jobs are available for the given allowList (22 ms)
19:15:25
✓ returns a job if it is in the allowList (55 ms)
19:15:25
✓ returns the most important job if it is in the allowList (51 ms)
19:15:25
✓ returns any job if filter is empty (55 ms)
19:15:25
✓ returns a new job when reporting job success (31 ms)
19:15:25
✓ returns a new job when reporting permanent error (38 ms)
19:15:25
✓ returns a new job when reporting retry-able error (27 ms)
19:15:25
✓ returns a new job when reporting progress if current one is cancelled (37 ms)
19:15:25
✓ returns a new job if job is already in progress elsewhere (1095 ms)
19:15:25
✓ avoids sending the same job to a new agent after a restart (1042 ms)
19:15:25
✓ avoids sending a completed job to a new agent after a restart (1052 ms)
19:15:25
✓ tracks job result if in progress (53 ms)
19:15:25
✓ tracks job result even if job is in queue (49 ms)
19:15:25
✓ ignores reported job error if unknown job (3 ms)
19:15:25
✓ ignores job result if unknown job (3 ms)
19:15:25 Timeouts
19:15:25
✓ tracks in progress jobs (1 ms)
19:15:25
✓ re-enqueues jobs that time out (102 ms)
19:15:25
✓ cancel stale jobs that time out (234 ms)
19:15:25
✓ keeps the jobs in progress while it is alive (203 ms)
19:15:25
✓ tracks in progress jobs (20 ms)
19:15:25
✓ re-enqueues jobs that time out (133 ms)
19:15:25
✓ cancel stale jobs that time out (275 ms)
19:15:25
✓ keeps the jobs in progress while it is alive (221 ms)
19:15:25 Retries
19:15:25
✓ retries jobs (2 ms)
19:15:25
✓ retries up to a maximum number of times (2 ms)
19:15:25
✓ passing retry=false does not retry (2 ms)
19:15:25
✓ does not retry if job is stale (27 ms)
19:15:25
✓ retries jobs (20 ms)
19:15:25
✓ retries up to a maximum number of times (25 ms)
19:15:25
✓ passing retry=false does not retry (24 ms)
19:15:25
✓ does not retry if job is stale (62 ms)
19:15:25 Database management
19:15:25
✓ re-enqueues proof requests on start (2 ms)
19:15:25
✓ restores proof results on start (1 ms)
19:15:25
✓ only re-enqueues unfinished jobs (1 ms)
19:15:25
✓ saves job when enqueued (2 ms)
19:15:25
✓ does not retain job if database fails to save (2 ms)
19:15:25
✓ saves job result (1 ms)
19:15:25
✓ saves result even if database fails to save (2 ms)
19:15:25
✓ saves job error (2 ms)
19:15:25
✓ saves job error even if database fails to save (2 ms)
19:15:25
✓ does not save job result if job is unknown (2 ms)
19:15:25
✓ does not save job error if job is unknown (2 ms)
19:15:25
✓ cleans up old jobs periodically (257 ms)
19:15:25
✓ re-enqueues proof requests on start (38 ms)
19:15:25
✓ restores proof results on start (50 ms)
19:15:25
✓ only re-enqueues unfinished jobs (44 ms)
19:15:25
✓ saves job when enqueued (19 ms)
19:15:25
✓ does not retain job if database fails to save (4 ms)
19:15:25
✓ saves job result (25 ms)
19:15:25
✓ saves result even if database fails to save (21 ms)
19:15:25
✓ saves job error (25 ms)
19:15:25
✓ saves job error even if database fails to save (22 ms)
19:15:25
✓ does not save job result if job is unknown (3 ms)
19:15:25
✓ does not save job error if job is unknown (3 ms)
19:15:25
✓ cleans up old jobs periodically (352 ms)
19:15:25
19:15:25
Test Suites: 1 passed, 1 total
19:15:25
Tests: 96 passed, 96 total
19:15:25
Snapshots: 0 total
19:15:25
Time: 11.348 s
19:15:25
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
19:15:25
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?