Parent Log:
http://ci.aztec-labs.com/f213c7a0e0587cf2
Command: 68e9475192d7aeac yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/f495236d35f9bb0d1086b7ec79256c2c2b7c98b8
Env: REF_NAME=gh-readonly-queue/staging/pr-15201-f0619dd82429a5973f3e1da8d7eb0877264908e3 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Sat Jun 21 13:20:13 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15201_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_staging
13:20:18 [13:20:18.243]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:096d0d9b0cf1a910 epochNumber=42
{"provingJobId":"1:BASE_PARITY:096d0d9b0cf1a910"}
13:20:18 [13:20:18.253]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:17635230b7a61752 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17635230b7a61752"}
13:20:18 [13:20:18.278]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b62b4488119f171 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b62b4488119f171"}
13:20:18 [13:20:18.280]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4d35e441316f0b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e4d35e441316f0b0"}
13:20:18 [13:20:18.281]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:313860753d284645 epochNumber=1
{"provingJobId":"1:BASE_PARITY:313860753d284645"}
13:20:18 [13:20:18.283]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:313860753d284645 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:313860753d284645"}
13:20:18 [13:20:18.285]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b72a820ba96de4c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b72a820ba96de4c3"}
13:20:18 [13:20:18.287]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b72a820ba96de4c3 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b72a820ba96de4c3"}
13:20:18 [13:20:18.287]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b72a820ba96de4c3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b72a820ba96de4c3"}
13:20:18 [13:20:18.288]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b72a820ba96de4c3 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b72a820ba96de4c3"}
13:20:18 [13:20:18.289]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.289]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9a1334bec20c5f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9a1334bec20c5f5"}
13:20:18 [13:20:18.321]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.322]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.323]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83026abb6592e42b epochNumber=1
{"provingJobId":"1:BASE_PARITY:83026abb6592e42b"}
13:20:18 [13:20:18.323]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:83026abb6592e42b
{"provingJobId":"1:BASE_PARITY:83026abb6592e42b"}
13:20:18 [13:20:18.324]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:83026abb6592e42b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:83026abb6592e42b"}
13:20:18 [13:20:18.325]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:83026abb6592e42b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:83026abb6592e42b"}
13:20:18 [13:20:18.327]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.327]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:122b47044bdd7625 epochNumber=1
{"provingJobId":"1:BASE_PARITY:122b47044bdd7625"}
13:20:18 [13:20:18.329]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:122b47044bdd7625
{"provingJobId":"1:BASE_PARITY:122b47044bdd7625"}
13:20:18 [13:20:18.330]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:122b47044bdd7625 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:122b47044bdd7625"}
13:20:18 [13:20:18.332]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.334]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9764471d026c0a01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9764471d026c0a01"}
13:20:18 [13:20:18.334]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9764471d026c0a01 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9764471d026c0a01"}
13:20:18 [13:20:18.335]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9764471d026c0a01 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9764471d026c0a01"}
13:20:18 [13:20:18.339]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.340]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc3485b64969c265 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc3485b64969c265"}
13:20:18 [13:20:18.341]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dc3485b64969c265 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dc3485b64969c265"}
13:20:18 [13:20:18.341]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dc3485b64969c265 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:dc3485b64969c265"}
13:20:18 [13:20:18.343]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.343]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd8cdc38c90d2b1c epochNumber=0
{"provingJobId":"1:BASE_PARITY:fd8cdc38c90d2b1c"}
13:20:18 [13:20:18.344]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fd8cdc38c90d2b1c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fd8cdc38c90d2b1c"}
13:20:18 [13:20:18.344]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0db5757a56b44a2 epochNumber=0
{"provingJobId":"1:BASE_PARITY:f0db5757a56b44a2"}
13:20:18 [13:20:18.344]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fd8cdc38c90d2b1c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fd8cdc38c90d2b1c"}
13:20:18 [13:20:18.345]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a880f90c3ab3611 epochNumber=0
{"provingJobId":"1:BASE_PARITY:8a880f90c3ab3611"}
13:20:18 [13:20:18.346]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.356]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61a89b9a60d658eb epochNumber=2
{"provingJobId":"1:BASE_PARITY:61a89b9a60d658eb"}
13:20:18 [13:20:18.358]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9345d4b78cc97a56 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9345d4b78cc97a56"}
13:20:18 [13:20:18.359]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.360]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:951549db6f7d4630 epochNumber=1
{"provingJobId":"1:BASE_PARITY:951549db6f7d4630"}
13:20:18 [13:20:18.361]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:feef205f75462831 epochNumber=1
{"provingJobId":"1:BASE_PARITY:feef205f75462831"}
13:20:18 [13:20:18.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5744f09c3a9471d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5744f09c3a9471d1"}
13:20:18 [13:20:18.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb2c6ee1ba2d5268 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fb2c6ee1ba2d5268"}
13:20:18 [13:20:18.363]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2eedd9cbf49d8fe9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2eedd9cbf49d8fe9"}
13:20:18 [13:20:18.364]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8fcc905bea1314c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8fcc905bea1314c1"}
13:20:18 [13:20:18.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6ae198285fe7691 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6ae198285fe7691"}
13:20:18 [13:20:18.367]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a76218f41b14cc12 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a76218f41b14cc12"}
13:20:18 [13:20:18.367]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f289afc9a0bccf58 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f289afc9a0bccf58"}
13:20:18 [13:20:18.369]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.369]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d5fac55f6956ac5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:d5fac55f6956ac5e"}
13:20:18 [13:20:18.370]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3289d065ef3317f7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3289d065ef3317f7"}
13:20:18 [13:20:18.370]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:665d1552f0d005a0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:665d1552f0d005a0"}
13:20:18 [13:20:18.370]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03a2291fc0ba7716 epochNumber=1
{"provingJobId":"1:BASE_PARITY:03a2291fc0ba7716"}
13:20:18 [13:20:18.372]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.372]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c001c908dc845c49 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c001c908dc845c49"}
13:20:18 [13:20:18.373]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a61ac7f68255b0b2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a61ac7f68255b0b2"}
13:20:18 [13:20:18.373]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c001c908dc845c49 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c001c908dc845c49"}
13:20:18 [13:20:18.374]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.375]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb4baeb52e4d404a epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb4baeb52e4d404a"}
13:20:18 [13:20:18.375]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e508543067d799b epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e508543067d799b"}
13:20:18 [13:20:18.375]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fb4baeb52e4d404a type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:fb4baeb52e4d404a"}
13:20:18 [13:20:18.376]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.377]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d2ca67d93ae105d epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d2ca67d93ae105d"}
13:20:18 [13:20:18.377]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d9d44516bdd8d2f epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d9d44516bdd8d2f"}
13:20:18 [13:20:18.377]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9d2ca67d93ae105d type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:9d2ca67d93ae105d"}
13:20:18 [13:20:18.378]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.380]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b896f91b5114590d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b896f91b5114590d"}
13:20:18 [13:20:18.381]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b896f91b5114590d
{"provingJobId":"1:BASE_PARITY:b896f91b5114590d"}
13:20:18 [13:20:18.381]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b896f91b5114590d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b896f91b5114590d"}
13:20:18 [13:20:18.382]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e0cd581674ebecf epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e0cd581674ebecf"}
13:20:18 [13:20:18.383]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b896f91b5114590d has already been completed
{"provingJobId":"1:BASE_PARITY:b896f91b5114590d"}
13:20:18 [13:20:18.384]
INFO:
prover-client:proving-broker Proving Broker started
13:20:18 [13:20:18.385]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9920730eee0c9f4c epochNumber=1
{"provingJobId":"1:BASE_PARITY:9920730eee0c9f4c"}
13:20:18 [13:20:18.385]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86b25c767d40c8ab epochNumber=2
{"provingJobId":"1:BASE_PARITY:86b25c767d40c8ab"}
13:20:19 [13:20:19.439]
INFO:
prover-client:proving-broker Proving Broker started
13:20:19 [13:20:19.440]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9920730eee0c9f4c settled=false
{"provingJobId":"1:BASE_PARITY:9920730eee0c9f4c","status":"pending"}
13:20:19 [13:20:19.440]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:86b25c767d40c8ab settled=false
{"provingJobId":"1:BASE_PARITY:86b25c767d40c8ab","status":"pending"}
13:20:19 [13:20:19.441]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9920730eee0c9f4c type=BASE_PARITY startedAt=1750512018386 older agent has taken job
{"provingJobId":"1:BASE_PARITY:9920730eee0c9f4c"}
13:20:19 [13:20:19.442]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9920730eee0c9f4c type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:9920730eee0c9f4c"}
13:20:19 [13:20:19.444]
INFO:
prover-client:proving-broker Proving Broker started
13:20:19 [13:20:19.444]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9fa8c112dab03c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a9fa8c112dab03c9"}
13:20:19 [13:20:19.445]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:38552ec60b536910 epochNumber=2
{"provingJobId":"1:BASE_PARITY:38552ec60b536910"}
13:20:20 [13:20:20.446]
INFO:
prover-client:proving-broker Proving Broker started
13:20:20 [13:20:20.447]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a9fa8c112dab03c9 settled=false
{"provingJobId":"1:BASE_PARITY:a9fa8c112dab03c9","status":"pending"}
13:20:20 [13:20:20.447]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:38552ec60b536910 settled=false
{"provingJobId":"1:BASE_PARITY:38552ec60b536910","status":"pending"}
13:20:20 [13:20:20.448]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a9fa8c112dab03c9 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:a9fa8c112dab03c9"}
13:20:20 [13:20:20.451]
INFO:
prover-client:proving-broker Proving Broker started
13:20:20 [13:20:20.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81e8756e4e2e7991 epochNumber=1
{"provingJobId":"1:BASE_PARITY:81e8756e4e2e7991"}
13:20:20 [13:20:20.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0b1f79ca726352d epochNumber=2
{"provingJobId":"1:BASE_PARITY:e0b1f79ca726352d"}
13:20:21 [13:20:21.453]
INFO:
prover-client:proving-broker Proving Broker started
13:20:21 [13:20:21.454]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:81e8756e4e2e7991 settled=false
{"provingJobId":"1:BASE_PARITY:81e8756e4e2e7991","status":"pending"}
13:20:21 [13:20:21.455]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e0b1f79ca726352d settled=false
{"provingJobId":"1:BASE_PARITY:e0b1f79ca726352d","status":"pending"}
13:20:21 [13:20:21.455]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:81e8756e4e2e7991 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:81e8756e4e2e7991"}
13:20:21 [13:20:21.456]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:81e8756e4e2e7991 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:81e8756e4e2e7991"}
13:20:21 [13:20:21.457]
INFO:
prover-client:proving-broker Proving Broker started
13:20:21 [13:20:21.457]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41ed52c3dc5a99bf epochNumber=1
{"provingJobId":"1:BASE_PARITY:41ed52c3dc5a99bf"}
13:20:21 [13:20:21.458]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfeae1674d0dffbb epochNumber=2
{"provingJobId":"1:BASE_PARITY:dfeae1674d0dffbb"}
13:20:21 [13:20:21.458]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:41ed52c3dc5a99bf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:41ed52c3dc5a99bf"}
13:20:21 [13:20:21.459]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dfeae1674d0dffbb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:dfeae1674d0dffbb"}
13:20:21 [13:20:21.460]
INFO:
prover-client:proving-broker Proving Broker started
13:20:21 [13:20:21.460]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fbcd1547cb48751f epochNumber=1
{"provingJobId":"1:BASE_PARITY:fbcd1547cb48751f"}
13:20:21 [13:20:21.460]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:790529a925316424 epochNumber=2
{"provingJobId":"1:BASE_PARITY:790529a925316424"}
13:20:21 [13:20:21.461]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fbcd1547cb48751f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fbcd1547cb48751f"}
13:20:21 [13:20:21.461]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fbcd1547cb48751f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fbcd1547cb48751f"}
13:20:21 [13:20:21.461]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:790529a925316424 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:790529a925316424"}
13:20:21 [13:20:21.462]
INFO:
prover-client:proving-broker Proving Broker started
13:20:21 [13:20:21.462]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2f18aecafc8d42e5 err=test error
{"provingJoId":"1:BASE_PARITY:2f18aecafc8d42e5"}
13:20:21 [13:20:21.463]
INFO:
prover-client:proving-broker Proving Broker started
13:20:21 [13:20:21.464]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7c96183e97d852c5 not found
{"provingJobId":"1:BASE_PARITY:7c96183e97d852c5"}
13:20:21 [13:20:21.465]
INFO:
prover-client:proving-broker Proving Broker started
13:20:21 [13:20:21.465]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d1d2e2b1354824a epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d1d2e2b1354824a"}
13:20:21 [13:20:21.467]
INFO:
prover-client:proving-broker Proving Broker started
13:20:21 [13:20:21.467]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04044cacae91ed64 epochNumber=1
{"provingJobId":"1:BASE_PARITY:04044cacae91ed64"}
13:20:21 [13:20:21.570]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:04044cacae91ed64 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:04044cacae91ed64"}
13:20:21 [13:20:21.576]
INFO:
prover-client:proving-broker Proving Broker started
13:20:21 [13:20:21.578]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59ea6d0b179a03c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:59ea6d0b179a03c4"}
13:20:21 [13:20:21.681]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:59ea6d0b179a03c4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:59ea6d0b179a03c4"}
13:20:21 [13:20:21.687]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2289b1a7d0b5d5b4 epochNumber=10
{"provingJobId":"1:BASE_PARITY:2289b1a7d0b5d5b4"}
13:20:21 [13:20:21.708]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:21 [13:20:21.814]
INFO:
prover-client:proving-broker Proving Broker started
13:20:21 [13:20:21.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97dc994d197f57c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:97dc994d197f57c4"}
13:20:21 [13:20:21.992]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:97dc994d197f57c4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:97dc994d197f57c4"}
13:20:22 [13:20:22.021]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.022]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a732bfd6fef5d62 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a732bfd6fef5d62"}
13:20:22 [13:20:22.023]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1a732bfd6fef5d62 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1a732bfd6fef5d62"}
13:20:22 [13:20:22.026]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.027]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:523aba9bd1989049 epochNumber=1
{"provingJobId":"1:BASE_PARITY:523aba9bd1989049"}
13:20:22 [13:20:22.028]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:523aba9bd1989049 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:523aba9bd1989049"}
13:20:22 [13:20:22.029]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:523aba9bd1989049 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:523aba9bd1989049"}
13:20:22 [13:20:22.030]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.031]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4fe2ea1b27c0f63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4fe2ea1b27c0f63"}
13:20:22 [13:20:22.032]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d4fe2ea1b27c0f63 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d4fe2ea1b27c0f63"}
13:20:22 [13:20:22.034]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.034]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f618f3a4ad3e384 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f618f3a4ad3e384"}
13:20:22 [13:20:22.035]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9f618f3a4ad3e384 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:9f618f3a4ad3e384"}
13:20:22 [13:20:22.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0f9c4f931f64701 epochNumber=3
{"provingJobId":"1:BASE_PARITY:c0f9c4f931f64701"}
13:20:22 [13:20:22.059]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:22 [13:20:22.061]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:9f618f3a4ad3e384 err=test error
{"provingJoId":"1:BASE_PARITY:9f618f3a4ad3e384"}
13:20:22 [13:20:22.063]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.063]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:531e5453645d1c33 settled=false
{"provingJobId":"1:BASE_PARITY:531e5453645d1c33","status":"pending"}
13:20:22 [13:20:22.064]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:78ee929b5f22f72d settled=false
{"provingJobId":"1:BASE_PARITY:78ee929b5f22f72d","status":"pending"}
13:20:22 [13:20:22.066]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.066]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d32493197f4d2bde settled=true
{"provingJobId":"1:BASE_PARITY:d32493197f4d2bde","status":"fulfilled"}
13:20:22 [13:20:22.066]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:8c5f448165e470d9 settled=true
{"provingJobId":"2:BASE_PARITY:8c5f448165e470d9","status":"fulfilled"}
13:20:22 [13:20:22.068]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.068]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0a7359526bc77574 settled=true
{"provingJobId":"1:BASE_PARITY:0a7359526bc77574","status":"fulfilled"}
13:20:22 [13:20:22.068]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3eb299e9936503cd settled=false
{"provingJobId":"1:BASE_PARITY:3eb299e9936503cd","status":"pending"}
13:20:22 [13:20:22.069]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.071]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb6e6e8c7be5fc03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb6e6e8c7be5fc03"}
13:20:22 [13:20:22.072]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:917a230b3bf62548 epochNumber=1
{"provingJobId":"1:BASE_PARITY:917a230b3bf62548"}
13:20:22 [13:20:22.073]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:917a230b3bf62548: Error: db error: Error: db error
13:20:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:917a230b3bf62548"}
13:20:22 [13:20:22.075]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.075]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe2f7d916bc409bf epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe2f7d916bc409bf"}
13:20:22 [13:20:22.075]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fe2f7d916bc409bf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fe2f7d916bc409bf"}
13:20:22 [13:20:22.076]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fe2f7d916bc409bf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fe2f7d916bc409bf"}
13:20:22 [13:20:22.077]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b6e63af475cc4b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b6e63af475cc4b0"}
13:20:22 [13:20:22.077]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b6e63af475cc4b0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7b6e63af475cc4b0"}
13:20:22 [13:20:22.078]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7b6e63af475cc4b0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7b6e63af475cc4b0"}
13:20:22 [13:20:22.078]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:7b6e63af475cc4b0: Error: db error
13:20:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:7b6e63af475cc4b0"}
13:20:22 [13:20:22.079]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.080]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d0ce6c5f1e2935e4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d0ce6c5f1e2935e4"}
13:20:22 [13:20:22.080]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d0ce6c5f1e2935e4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d0ce6c5f1e2935e4"}
13:20:22 [13:20:22.080]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d0ce6c5f1e2935e4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d0ce6c5f1e2935e4"}
13:20:22 [13:20:22.081]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4df5618f1fce2d97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4df5618f1fce2d97"}
13:20:22 [13:20:22.082]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4df5618f1fce2d97 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4df5618f1fce2d97"}
13:20:22 [13:20:22.082]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4df5618f1fce2d97 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4df5618f1fce2d97"}
13:20:22 [13:20:22.083]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:4df5618f1fce2d97 jobErr=test error: Error: db error
13:20:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:4df5618f1fce2d97"}
13:20:22 [13:20:22.084]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.084]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:56d07acb458588c5 not found
{"provingJobId":"1:BASE_PARITY:56d07acb458588c5"}
13:20:22 [13:20:22.088]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.089]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c1ca7276fcdfa9fd err=test error
{"provingJoId":"1:BASE_PARITY:c1ca7276fcdfa9fd"}
13:20:22 [13:20:22.091]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.102]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13ccfce642dd8252 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13ccfce642dd8252"}
13:20:22 [13:20:22.102]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:13ccfce642dd8252 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:13ccfce642dd8252"}
13:20:22 [13:20:22.103]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:13ccfce642dd8252 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:13ccfce642dd8252"}
13:20:22 [13:20:22.114]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:8d22b9de0fb9e0b2 epochNumber=2
{"provingJobId":"2:BASE_PARITY:8d22b9de0fb9e0b2"}
13:20:22 [13:20:22.115]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:8d22b9de0fb9e0b2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:8d22b9de0fb9e0b2"}
13:20:22 [13:20:22.115]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:8d22b9de0fb9e0b2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:8d22b9de0fb9e0b2"}
13:20:22 [13:20:22.176]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:e22361b136e76c32 epochNumber=3
{"provingJobId":"3:BASE_PARITY:e22361b136e76c32"}
13:20:22 [13:20:22.193]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:22 [13:20:22.238]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:51c6b0843d157048 epochNumber=4
{"provingJobId":"4:BASE_PARITY:51c6b0843d157048"}
13:20:22 [13:20:22.247]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:22 [13:20:22.301]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:e21ae7344d9a4ef5 epochNumber=5
{"provingJobId":"5:BASE_PARITY:e21ae7344d9a4ef5"}
13:20:22 [13:20:22.325]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:22 [13:20:22.357]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.358]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ff314ede11254e5 epochNumber=42
{"provingJobId":"1:BASE_PARITY:9ff314ede11254e5"}
13:20:22 [13:20:22.360]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testU5qmiM/42 with map size 1073741824
13:20:22 [13:20:22.365]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.398]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:15bae9a19e696fcf epochNumber=1
{"provingJobId":"1:BASE_PARITY:15bae9a19e696fcf"}
13:20:22 [13:20:22.405]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.406]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ec617c29aeaa919 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3ec617c29aeaa919"}
13:20:22 [13:20:22.407]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbdAzZT/1 with map size 1073741824
13:20:22 [13:20:22.412]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.441]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c897071c5eb5142d epochNumber=1
{"provingJobId":"1:BASE_PARITY:c897071c5eb5142d"}
13:20:22 [13:20:22.451]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:627dd56fab7668fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:627dd56fab7668fe"}
13:20:22 [13:20:22.452]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmw2Tko/1 with map size 1073741824
13:20:22 [13:20:22.455]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.478]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:627dd56fab7668fe epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:627dd56fab7668fe"}
13:20:22 [13:20:22.482]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.483]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dabdbaea36e641ce epochNumber=1
{"provingJobId":"1:BASE_PARITY:dabdbaea36e641ce"}
13:20:22 [13:20:22.483]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoZrml7/1 with map size 1073741824
13:20:22 [13:20:22.485]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.503]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dabdbaea36e641ce epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dabdbaea36e641ce"}
13:20:22 [13:20:22.504]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dabdbaea36e641ce type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dabdbaea36e641ce"}
13:20:22 [13:20:22.511]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dabdbaea36e641ce epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dabdbaea36e641ce"}
13:20:22 [13:20:22.518]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.519]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d0efb66781f32e5a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d0efb66781f32e5a"}
13:20:22 [13:20:22.520]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJzggge/1 with map size 1073741824
13:20:22 [13:20:22.522]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.554]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.558]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.559]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef1723e5edeb1a10 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef1723e5edeb1a10"}
13:20:22 [13:20:22.559]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeq8GQD/1 with map size 1073741824
13:20:22 [13:20:22.561]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.581]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ef1723e5edeb1a10
{"provingJobId":"1:BASE_PARITY:ef1723e5edeb1a10"}
13:20:22 [13:20:22.582]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ef1723e5edeb1a10 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ef1723e5edeb1a10"}
13:20:22 [13:20:22.582]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ef1723e5edeb1a10 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ef1723e5edeb1a10"}
13:20:22 [13:20:22.595]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.596]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6dcac21b435b787 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b6dcac21b435b787"}
13:20:22 [13:20:22.597]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcsBPLL/1 with map size 1073741824
13:20:22 [13:20:22.599]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.627]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b6dcac21b435b787
{"provingJobId":"1:BASE_PARITY:b6dcac21b435b787"}
13:20:22 [13:20:22.628]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b6dcac21b435b787 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b6dcac21b435b787"}
13:20:22 [13:20:22.637]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.637]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:803c25ad9f8f4213 epochNumber=1
{"provingJobId":"1:BASE_PARITY:803c25ad9f8f4213"}
13:20:22 [13:20:22.641]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7PE6d1/1 with map size 1073741824
13:20:22 [13:20:22.647]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.664]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:803c25ad9f8f4213 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:803c25ad9f8f4213"}
13:20:22 [13:20:22.664]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:803c25ad9f8f4213 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:803c25ad9f8f4213"}
13:20:22 [13:20:22.672]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.673]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48bcb0ea6eaba016 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48bcb0ea6eaba016"}
13:20:22 [13:20:22.673]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGoI3fL/1 with map size 1073741824
13:20:22 [13:20:22.678]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.695]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:48bcb0ea6eaba016 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:48bcb0ea6eaba016"}
13:20:22 [13:20:22.695]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:48bcb0ea6eaba016 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:48bcb0ea6eaba016"}
13:20:22 [13:20:22.703]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.704]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0742aa6bcba77db5 epochNumber=0
{"provingJobId":"1:BASE_PARITY:0742aa6bcba77db5"}
13:20:22 [13:20:22.705]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testEo6QH4/0 with map size 1073741824
13:20:22 [13:20:22.706]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.726]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0742aa6bcba77db5 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0742aa6bcba77db5"}
13:20:22 [13:20:22.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:27ae48508d876fbe epochNumber=0
{"provingJobId":"1:BASE_PARITY:27ae48508d876fbe"}
13:20:22 [13:20:22.728]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0742aa6bcba77db5 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0742aa6bcba77db5"}
13:20:22 [13:20:22.728]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:470464a717f6ad0f epochNumber=0
{"provingJobId":"1:BASE_PARITY:470464a717f6ad0f"}
13:20:22 [13:20:22.741]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.743]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.744]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:894d1dc78597f362 epochNumber=2
{"provingJobId":"1:BASE_PARITY:894d1dc78597f362"}
13:20:22 [13:20:22.744]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testEFkApr/2 with map size 1073741824
13:20:22 [13:20:22.746]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.760]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e47b5502904ac048 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e47b5502904ac048"}
13:20:22 [13:20:22.762]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEFkApr/1 with map size 1073741824
13:20:22 [13:20:22.764]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.784]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f67345c3ef938329 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f67345c3ef938329"}
13:20:22 [13:20:22.788]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjRVicI/1 with map size 1073741824
13:20:22 [13:20:22.791]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.816]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.817]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a3e6be0484ea2d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a3e6be0484ea2d5"}
13:20:22 [13:20:22.818]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testttlWhN/1 with map size 1073741824
13:20:22 [13:20:22.822]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.845]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc44912e079c2c1e epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc44912e079c2c1e"}
13:20:22 [13:20:22.851]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f243c8a025e25e5a epochNumber=2
{"provingJobId":"1:BASE_PARITY:f243c8a025e25e5a"}
13:20:22 [13:20:22.853]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testttlWhN/2 with map size 1073741824
13:20:22 [13:20:22.855]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.872]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:134df83f3e036892 epochNumber=1
{"provingJobId":"1:BASE_PARITY:134df83f3e036892"}
13:20:22 [13:20:22.881]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.882]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b926a773a69c7082 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b926a773a69c7082"}
13:20:22 [13:20:22.882]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXFqVc0/1 with map size 1073741824
13:20:22 [13:20:22.885]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.905]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:180c9ca1f44fd531 epochNumber=1
{"provingJobId":"1:BASE_PARITY:180c9ca1f44fd531"}
13:20:22 [13:20:22.911]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e86d0c5782e250d epochNumber=2
{"provingJobId":"1:BASE_PARITY:5e86d0c5782e250d"}
13:20:22 [13:20:22.912]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testXFqVc0/2 with map size 1073741824
13:20:22 [13:20:22.917]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.937]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53237a16c0b4e987 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53237a16c0b4e987"}
13:20:22 [13:20:22.947]
INFO:
prover-client:proving-broker Proving Broker started
13:20:22 [13:20:22.948]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31c751f65df9f069 epochNumber=1
{"provingJobId":"1:BASE_PARITY:31c751f65df9f069"}
13:20:22 [13:20:22.949]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testER9nnb/1 with map size 1073741824
13:20:22 [13:20:22.950]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.967]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7433221bec9fc751 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7433221bec9fc751"}
13:20:22 [13:20:22.973]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22683b667d2ecfb5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:22683b667d2ecfb5"}
13:20:22 [13:20:22.974]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testER9nnb/2 with map size 1073741824
13:20:22 [13:20:22.976]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:22 [13:20:22.992]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f58eb0a1f380fb62 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f58eb0a1f380fb62"}
13:20:23 [13:20:23.002]
INFO:
prover-client:proving-broker Proving Broker started
13:20:23 [13:20:23.003]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d55da20f676bf6e4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d55da20f676bf6e4"}
13:20:23 [13:20:23.004]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8n4NX7/1 with map size 1073741824
13:20:23 [13:20:23.006]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:23 [13:20:23.026]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e3302e0eef169d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e3302e0eef169d5"}
13:20:23 [13:20:23.033]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d55da20f676bf6e4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d55da20f676bf6e4"}
13:20:23 [13:20:23.045]
INFO:
prover-client:proving-broker Proving Broker started
13:20:23 [13:20:23.046]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab2b98625a18b736 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab2b98625a18b736"}
13:20:23 [13:20:23.047]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFFLOjL/1 with map size 1073741824
13:20:23 [13:20:23.049]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:23 [13:20:23.067]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d387cd2301e996e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d387cd2301e996e9"}
13:20:23 [13:20:23.073]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ab2b98625a18b736 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:ab2b98625a18b736"}
13:20:23 [13:20:23.082]
INFO:
prover-client:proving-broker Proving Broker started
13:20:23 [13:20:23.083]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a4ff6a3c41b4ac9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a4ff6a3c41b4ac9"}
13:20:23 [13:20:23.084]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testq8E2ho/1 with map size 1073741824
13:20:23 [13:20:23.085]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:23 [13:20:23.103]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f7e6c1b28f597fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f7e6c1b28f597fe"}
13:20:23 [13:20:23.108]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1a4ff6a3c41b4ac9 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:1a4ff6a3c41b4ac9"}
13:20:23 [13:20:23.112]
INFO:
prover-client:proving-broker Proving Broker started
13:20:23 [13:20:23.113]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f0e608ca3d0cb7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f0e608ca3d0cb7d"}
13:20:23 [13:20:23.114]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxkYzyU/1 with map size 1073741824
13:20:23 [13:20:23.116]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:23 [13:20:23.142]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3f0e608ca3d0cb7d
{"provingJobId":"1:BASE_PARITY:3f0e608ca3d0cb7d"}
13:20:23 [13:20:23.142]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3f0e608ca3d0cb7d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3f0e608ca3d0cb7d"}
13:20:23 [13:20:23.148]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4d98d74280812a9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4d98d74280812a9"}
13:20:23 [13:20:23.154]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3f0e608ca3d0cb7d has already been completed
{"provingJobId":"1:BASE_PARITY:3f0e608ca3d0cb7d"}
13:20:23 [13:20:23.158]
INFO:
prover-client:proving-broker Proving Broker started
13:20:23 [13:20:23.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b94babb7aa508e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b94babb7aa508e2"}
13:20:23 [13:20:23.159]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjBOqus/1 with map size 1073741824
13:20:23 [13:20:23.161]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:23 [13:20:23.177]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:867a6eeb7133a6b4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:867a6eeb7133a6b4"}
13:20:23 [13:20:23.178]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjBOqus/2 with map size 1073741824
13:20:23 [13:20:23.180]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:24 [13:20:24.249]
INFO:
prover-client:proving-broker Proving Broker started
13:20:24 [13:20:24.257]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0b94babb7aa508e2 settled=false
{"provingJobId":"1:BASE_PARITY:0b94babb7aa508e2","status":"pending"}
13:20:24 [13:20:24.259]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:867a6eeb7133a6b4 settled=false
{"provingJobId":"1:BASE_PARITY:867a6eeb7133a6b4","status":"pending"}
13:20:24 [13:20:24.260]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0b94babb7aa508e2 type=BASE_PARITY startedAt=1750512023197 older agent has taken job
{"provingJobId":"1:BASE_PARITY:0b94babb7aa508e2"}
13:20:24 [13:20:24.261]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0b94babb7aa508e2 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:0b94babb7aa508e2"}
13:20:24 [13:20:24.264]
INFO:
prover-client:proving-broker Proving Broker started
13:20:24 [13:20:24.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13e5c832298d732f epochNumber=1
{"provingJobId":"1:BASE_PARITY:13e5c832298d732f"}
13:20:24 [13:20:24.266]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7uigSX/1 with map size 1073741824
13:20:24 [13:20:24.268]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:24 [13:20:24.288]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95ac0bf4e29520d8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:95ac0bf4e29520d8"}
13:20:24 [13:20:24.289]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7uigSX/2 with map size 1073741824
13:20:24 [13:20:24.292]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:25 [13:20:25.311]
INFO:
prover-client:proving-broker Proving Broker started
13:20:25 [13:20:25.315]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:13e5c832298d732f settled=false
{"provingJobId":"1:BASE_PARITY:13e5c832298d732f","status":"pending"}
13:20:25 [13:20:25.319]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:95ac0bf4e29520d8 settled=false
{"provingJobId":"1:BASE_PARITY:95ac0bf4e29520d8","status":"pending"}
13:20:25 [13:20:25.320]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:13e5c832298d732f type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:13e5c832298d732f"}
13:20:25 [13:20:25.325]
INFO:
prover-client:proving-broker Proving Broker started
13:20:25 [13:20:25.326]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7c627af9c93a872 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7c627af9c93a872"}
13:20:25 [13:20:25.331]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcKcD7w/1 with map size 1073741824
13:20:25 [13:20:25.335]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:25 [13:20:25.358]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de3191ace4335dff epochNumber=2
{"provingJobId":"1:BASE_PARITY:de3191ace4335dff"}
13:20:25 [13:20:25.359]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcKcD7w/2 with map size 1073741824
13:20:25 [13:20:25.363]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:26 [13:20:26.384]
INFO:
prover-client:proving-broker Proving Broker started
13:20:26 [13:20:26.386]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b7c627af9c93a872 settled=false
{"provingJobId":"1:BASE_PARITY:b7c627af9c93a872","status":"pending"}
13:20:26 [13:20:26.388]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:de3191ace4335dff settled=false
{"provingJobId":"1:BASE_PARITY:de3191ace4335dff","status":"pending"}
13:20:26 [13:20:26.390]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b7c627af9c93a872 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b7c627af9c93a872"}
13:20:26 [13:20:26.390]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b7c627af9c93a872 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b7c627af9c93a872"}
13:20:26 [13:20:26.400]
INFO:
prover-client:proving-broker Proving Broker started
13:20:26 [13:20:26.401]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be5a544832011b3f epochNumber=1
{"provingJobId":"1:BASE_PARITY:be5a544832011b3f"}
13:20:26 [13:20:26.402]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMbny1Y/1 with map size 1073741824
13:20:26 [13:20:26.403]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:26 [13:20:26.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e968ae057c061051 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e968ae057c061051"}
13:20:26 [13:20:26.438]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMbny1Y/2 with map size 1073741824
13:20:26 [13:20:26.440]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:26 [13:20:26.457]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:be5a544832011b3f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:be5a544832011b3f"}
13:20:26 [13:20:26.463]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e968ae057c061051 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e968ae057c061051"}
13:20:26 [13:20:26.474]
INFO:
prover-client:proving-broker Proving Broker started
13:20:26 [13:20:26.476]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a767dde382c9a994 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a767dde382c9a994"}
13:20:26 [13:20:26.477]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEPRh1R/1 with map size 1073741824
13:20:26 [13:20:26.479]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:26 [13:20:26.496]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cfd4a6636bf004e2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cfd4a6636bf004e2"}
13:20:26 [13:20:26.497]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testEPRh1R/2 with map size 1073741824
13:20:26 [13:20:26.499]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:26 [13:20:26.520]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a767dde382c9a994 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a767dde382c9a994"}
13:20:26 [13:20:26.520]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a767dde382c9a994 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a767dde382c9a994"}
13:20:26 [13:20:26.531]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cfd4a6636bf004e2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:cfd4a6636bf004e2"}
13:20:26 [13:20:26.543]
INFO:
prover-client:proving-broker Proving Broker started
13:20:26 [13:20:26.545]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f6d5b343c90d6b15 err=test error
{"provingJoId":"1:BASE_PARITY:f6d5b343c90d6b15"}
13:20:26 [13:20:26.548]
INFO:
prover-client:proving-broker Proving Broker started
13:20:26 [13:20:26.549]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e78f8204ef20f44d not found
{"provingJobId":"1:BASE_PARITY:e78f8204ef20f44d"}
13:20:26 [13:20:26.552]
INFO:
prover-client:proving-broker Proving Broker started
13:20:26 [13:20:26.553]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7dd2d8a5ef837d06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7dd2d8a5ef837d06"}
13:20:26 [13:20:26.554]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUHdwF4/1 with map size 1073741824
13:20:26 [13:20:26.556]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:26 [13:20:26.581]
INFO:
prover-client:proving-broker Proving Broker started
13:20:26 [13:20:26.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79dc38d99d252e00 epochNumber=1
{"provingJobId":"1:BASE_PARITY:79dc38d99d252e00"}
13:20:26 [13:20:26.583]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5ukD6B/1 with map size 1073741824
13:20:26 [13:20:26.585]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:26 [13:20:26.709]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:79dc38d99d252e00 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:79dc38d99d252e00"}
13:20:26 [13:20:26.721]
INFO:
prover-client:proving-broker Proving Broker started
13:20:26 [13:20:26.723]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a7430c7854e4530 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a7430c7854e4530"}
13:20:26 [13:20:26.724]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testz2Gnld/1 with map size 1073741824
13:20:26 [13:20:26.734]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:26 [13:20:26.877]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1a7430c7854e4530 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1a7430c7854e4530"}
13:20:26 [13:20:26.878]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1ffc2a60c4553d5 epochNumber=10
{"provingJobId":"1:BASE_PARITY:d1ffc2a60c4553d5"}
13:20:26 [13:20:26.880]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testz2Gnld/10 with map size 1073741824
13:20:26 [13:20:26.883]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:26 [13:20:26.903]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:26 [13:20:26.903]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:20:27 [13:20:27.035]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7cdc7c4bccc2021e epochNumber=1
{"provingJobId":"1:BASE_PARITY:7cdc7c4bccc2021e"}
13:20:27 [13:20:27.039]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoU65MY/1 with map size 1073741824
13:20:27 [13:20:27.041]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.215]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7cdc7c4bccc2021e timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:7cdc7c4bccc2021e"}
13:20:27 [13:20:27.267]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:511127c3aa4d8134 epochNumber=1
{"provingJobId":"1:BASE_PARITY:511127c3aa4d8134"}
13:20:27 [13:20:27.270]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQT2uQ2/1 with map size 1073741824
13:20:27 [13:20:27.271]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.289]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:511127c3aa4d8134 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:511127c3aa4d8134"}
13:20:27 [13:20:27.296]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.297]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd15c49f5ebb91b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd15c49f5ebb91b8"}
13:20:27 [13:20:27.298]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZgOuYr/1 with map size 1073741824
13:20:27 [13:20:27.299]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.320]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:dd15c49f5ebb91b8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:dd15c49f5ebb91b8"}
13:20:27 [13:20:27.321]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dd15c49f5ebb91b8 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:dd15c49f5ebb91b8"}
13:20:27 [13:20:27.333]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.334]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b11326a4f3b9455e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b11326a4f3b9455e"}
13:20:27 [13:20:27.335]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhCvmET/1 with map size 1073741824
13:20:27 [13:20:27.337]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.363]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b11326a4f3b9455e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b11326a4f3b9455e"}
13:20:27 [13:20:27.373]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.374]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:969e4a3d9bbb6aba epochNumber=1
{"provingJobId":"1:BASE_PARITY:969e4a3d9bbb6aba"}
13:20:27 [13:20:27.374]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjP9hf2/1 with map size 1073741824
13:20:27 [13:20:27.377]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.395]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:969e4a3d9bbb6aba type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:969e4a3d9bbb6aba"}
13:20:27 [13:20:27.397]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6cc63bc7028081f8 epochNumber=3
{"provingJobId":"1:BASE_PARITY:6cc63bc7028081f8"}
13:20:27 [13:20:27.398]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testjP9hf2/3 with map size 1073741824
13:20:27 [13:20:27.402]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.449]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:27 [13:20:27.450]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:20:27 [13:20:27.450]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:969e4a3d9bbb6aba err=test error
{"provingJoId":"1:BASE_PARITY:969e4a3d9bbb6aba"}
13:20:27 [13:20:27.457]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testE41m9Y/1 with map size 1073741824
13:20:27 [13:20:27.460]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.477]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testE41m9Y/2 with map size 1073741824
13:20:27 [13:20:27.479]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.494]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.496]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:60f60c80955e3250 settled=false
{"provingJobId":"1:BASE_PARITY:60f60c80955e3250","status":"pending"}
13:20:27 [13:20:27.497]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9dc060155d96e331 settled=false
{"provingJobId":"1:BASE_PARITY:9dc060155d96e331","status":"pending"}
13:20:27 [13:20:27.502]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKKc2EM/1 with map size 1073741824
13:20:27 [13:20:27.504]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.520]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKKc2EM/2 with map size 1073741824
13:20:27 [13:20:27.523]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.549]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.551]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c89c6afcc852276d settled=true
{"provingJobId":"1:BASE_PARITY:c89c6afcc852276d","status":"fulfilled"}
13:20:27 [13:20:27.555]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:19133e900597eb71 settled=true
{"provingJobId":"2:BASE_PARITY:19133e900597eb71","status":"fulfilled"}
13:20:27 [13:20:27.560]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIaTNez/1 with map size 1073741824
13:20:27 [13:20:27.562]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.584]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIaTNez/2 with map size 1073741824
13:20:27 [13:20:27.587]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.607]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.614]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5dfab48b51eae153 settled=true
{"provingJobId":"1:BASE_PARITY:5dfab48b51eae153","status":"fulfilled"}
13:20:27 [13:20:27.616]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2e72295627f91942 settled=false
{"provingJobId":"1:BASE_PARITY:2e72295627f91942","status":"pending"}
13:20:27 [13:20:27.622]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.623]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50561528216ad7bf epochNumber=1
{"provingJobId":"1:BASE_PARITY:50561528216ad7bf"}
13:20:27 [13:20:27.624]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSRDe5r/1 with map size 1073741824
13:20:27 [13:20:27.630]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.655]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.656]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4a3f4f7b782d522 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4a3f4f7b782d522"}
13:20:27 [13:20:27.656]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:f4a3f4f7b782d522: Error: db error: Error: db error
13:20:27 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:f4a3f4f7b782d522"}
13:20:27 [13:20:27.660]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.660]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de6a76d5e6204b48 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de6a76d5e6204b48"}
13:20:27 [13:20:27.662]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSCuzOj/1 with map size 1073741824
13:20:27 [13:20:27.664]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.682]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:de6a76d5e6204b48 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:de6a76d5e6204b48"}
13:20:27 [13:20:27.683]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:de6a76d5e6204b48 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:de6a76d5e6204b48"}
13:20:27 [13:20:27.694]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.695]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ac02c662478e5ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ac02c662478e5ca"}
13:20:27 [13:20:27.696]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testS6izlI/1 with map size 1073741824
13:20:27 [13:20:27.698]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.720]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5ac02c662478e5ca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5ac02c662478e5ca"}
13:20:27 [13:20:27.720]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5ac02c662478e5ca type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5ac02c662478e5ca"}
13:20:27 [13:20:27.721]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:5ac02c662478e5ca: Error: db error
13:20:27 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:5ac02c662478e5ca"}
13:20:27 [13:20:27.726]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:792f8ce69d657898 epochNumber=1
{"provingJobId":"1:BASE_PARITY:792f8ce69d657898"}
13:20:27 [13:20:27.728]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYgqz75/1 with map size 1073741824
13:20:27 [13:20:27.731]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.748]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:792f8ce69d657898 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:792f8ce69d657898"}
13:20:27 [13:20:27.749]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:792f8ce69d657898 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:792f8ce69d657898"}
13:20:27 [13:20:27.763]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.764]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:866e37319e3f1a87 epochNumber=1
{"provingJobId":"1:BASE_PARITY:866e37319e3f1a87"}
13:20:27 [13:20:27.766]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDLR5Cb/1 with map size 1073741824
13:20:27 [13:20:27.769]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.787]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:866e37319e3f1a87 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:866e37319e3f1a87"}
13:20:27 [13:20:27.788]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:866e37319e3f1a87 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:866e37319e3f1a87"}
13:20:27 [13:20:27.788]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:866e37319e3f1a87 jobErr=test error: Error: db error
13:20:27 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:866e37319e3f1a87"}
13:20:27 [13:20:27.796]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.797]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:49d03834632013e4 not found
{"provingJobId":"1:BASE_PARITY:49d03834632013e4"}
13:20:27 [13:20:27.800]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.801]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:71d70b3854714f75 err=test error
{"provingJoId":"1:BASE_PARITY:71d70b3854714f75"}
13:20:27 [13:20:27.806]
INFO:
prover-client:proving-broker Proving Broker started
13:20:27 [13:20:27.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d26d3603e8fc34b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d26d3603e8fc34b"}
13:20:27 [13:20:27.817]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpkJuds/1 with map size 1073741824
13:20:27 [13:20:27.825]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.844]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4d26d3603e8fc34b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4d26d3603e8fc34b"}
13:20:27 [13:20:27.844]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4d26d3603e8fc34b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4d26d3603e8fc34b"}
13:20:27 [13:20:27.862]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:a3aa82163937654a epochNumber=2
{"provingJobId":"2:BASE_PARITY:a3aa82163937654a"}
13:20:27 [13:20:27.863]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpkJuds/2 with map size 1073741824
13:20:27 [13:20:27.867]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.885]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:a3aa82163937654a type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:a3aa82163937654a"}
13:20:27 [13:20:27.886]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:a3aa82163937654a type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:a3aa82163937654a"}
13:20:27 [13:20:27.953]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:b7dd59acfa26e3eb epochNumber=3
{"provingJobId":"3:BASE_PARITY:b7dd59acfa26e3eb"}
13:20:27 [13:20:27.955]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testpkJuds/3 with map size 1073741824
13:20:27 [13:20:27.958]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:27 [13:20:27.984]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:27 [13:20:27.985]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:20:28 [13:20:28.044]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:e46865bd3639b054 epochNumber=4
{"provingJobId":"4:BASE_PARITY:e46865bd3639b054"}
13:20:28 [13:20:28.045]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testpkJuds/4 with map size 1073741824
13:20:28 [13:20:28.050]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:28 [13:20:28.093]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:28 [13:20:28.094]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
13:20:28 [13:20:28.133]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:b11dfcd88902d206 epochNumber=5
{"provingJobId":"5:BASE_PARITY:b11dfcd88902d206"}
13:20:28 [13:20:28.134]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testpkJuds/5 with map size 1073741824
13:20:28 [13:20:28.137]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:20:28 [13:20:28.173]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:20:28 [13:20:28.173]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
13:20:28
PASS src/proving_broker/proving_broker.test.ts (
13.402 s)
13:20:28 ProvingBroker
13:20:28 Producer API
13:20:28
✓ refuses stale jobs (38 ms)
13:20:28
✓ enqueues jobs (4 ms)
13:20:28
✓ ignores duplicate jobs (4 ms)
13:20:28
✓ reports correct status when enqueuing repeat jobs (3 ms)
13:20:28
✓ throws an error in case of duplicate job IDs (30 ms)
13:20:28
✓ returns not-found status for non-existing jobs (2 ms)
13:20:28
✓ cancels jobs in queue (4 ms)
13:20:28
✓ cancels jobs in-progress (5 ms)
13:20:28
✓ returns job result if successful (7 ms)
13:20:28
✓ returns job error if failed (4 ms)
13:20:28
✓ correctly returns job status for concurrent writes (3 ms)
13:20:28
✓ refuses stale jobs (48 ms)
13:20:28
✓ enqueues jobs (46 ms)
13:20:28
✓ ignores duplicate jobs (31 ms)
13:20:28
✓ reports correct status when enqueuing repeat jobs (31 ms)
13:20:28
✓ throws an error in case of duplicate job IDs (39 ms)
13:20:28
✓ returns not-found status for non-existing jobs (3 ms)
13:20:28
✓ cancels jobs in queue (35 ms)
13:20:28
✓ cancels jobs in-progress (43 ms)
13:20:28
✓ returns job result if successful (36 ms)
13:20:28
✓ returns job error if failed (31 ms)
13:20:28
✓ correctly returns job status for concurrent writes (38 ms)
13:20:28 Consumer API
13:20:28
✓ returns undefined if no jobs are available (9 ms)
13:20:28
✓ returns jobs in priority order (3 ms)
13:20:28
✓ returns undefined if no jobs are available for the given allowList (1 ms)
13:20:28
✓ returns a job if it is in the allowList (3 ms)
13:20:28
✓ returns the most important job if it is in the allowList (4 ms)
13:20:28
✓ returns any job if filter is empty (3 ms)
13:20:28
✓ returns a new job when reporting job success (3 ms)
13:20:28
✓ returns a new job when reporting permanent error (2 ms)
13:20:28
✓ returns a new job when reporting retry-able error (2 ms)
13:20:28
✓ returns a new job when reporting progress if current one is cancelled (5 ms)
13:20:28
✓ returns a new job if job is already in progress elsewhere (1059 ms)
13:20:28
✓ avoids sending the same job to a new agent after a restart (1007 ms)
13:20:28
✓ avoids sending a completed job to a new agent after a restart (1006 ms)
13:20:28
✓ tracks job result if in progress (2 ms)
13:20:28
✓ tracks job result even if job is in queue (3 ms)
13:20:28
✓ ignores reported job error if unknown job (1 ms)
13:20:28
✓ ignores job result if unknown job (1 ms)
13:20:28
✓ returns undefined if no jobs are available (2 ms)
13:20:28
✓ returns jobs in priority order (39 ms)
13:20:28
✓ returns undefined if no jobs are available for the given allowList (32 ms)
13:20:28
✓ returns a job if it is in the allowList (64 ms)
13:20:28
✓ returns the most important job if it is in the allowList (65 ms)
13:20:28
✓ returns any job if filter is empty (54 ms)
13:20:28
✓ returns a new job when reporting job success (41 ms)
13:20:28
✓ returns a new job when reporting permanent error (39 ms)
13:20:28
✓ returns a new job when reporting retry-able error (29 ms)
13:20:28
✓ returns a new job when reporting progress if current one is cancelled (45 ms)
13:20:28
✓ returns a new job if job is already in progress elsewhere (1107 ms)
13:20:28
✓ avoids sending the same job to a new agent after a restart (1060 ms)
13:20:28
✓ avoids sending a completed job to a new agent after a restart (1075 ms)
13:20:28
✓ tracks job result if in progress (73 ms)
13:20:28
✓ tracks job result even if job is in queue (69 ms)
13:20:28
✓ ignores reported job error if unknown job (5 ms)
13:20:28
✓ ignores job result if unknown job (4 ms)
13:20:28 Timeouts
13:20:28
✓ tracks in progress jobs (1 ms)
13:20:28
✓ re-enqueues jobs that time out (109 ms)
13:20:28
✓ cancel stale jobs that time out (237 ms)
13:20:28
✓ keeps the jobs in progress while it is alive (207 ms)
13:20:28
✓ tracks in progress jobs (29 ms)
13:20:28
✓ re-enqueues jobs that time out (137 ms)
13:20:28
✓ cancel stale jobs that time out (312 ms)
13:20:28
✓ keeps the jobs in progress while it is alive (235 ms)
13:20:28 Retries
13:20:28
✓ retries jobs (4 ms)
13:20:28
✓ retries up to a maximum number of times (5 ms)
13:20:28
✓ passing retry=false does not retry (3 ms)
13:20:28
✓ does not retry if job is stale (28 ms)
13:20:28
✓ retries jobs (28 ms)
13:20:28
✓ retries up to a maximum number of times (36 ms)
13:20:28
✓ passing retry=false does not retry (40 ms)
13:20:28
✓ does not retry if job is stale (82 ms)
13:20:28 Database management
13:20:28
✓ re-enqueues proof requests on start (3 ms)
13:20:28
✓ restores proof results on start (2 ms)
13:20:28
✓ only re-enqueues unfinished jobs (2 ms)
13:20:28
✓ saves job when enqueued (3 ms)
13:20:28
✓ does not retain job if database fails to save (2 ms)
13:20:28
✓ saves job result (2 ms)
13:20:28
✓ saves result even if database fails to save (3 ms)
13:20:28
✓ saves job error (2 ms)
13:20:28
✓ saves job error even if database fails to save (2 ms)
13:20:28
✓ does not save job result if job is unknown (2 ms)
13:20:28
✓ does not save job error if job is unknown (3 ms)
13:20:28
✓ cleans up old jobs periodically (263 ms)
13:20:28
✓ re-enqueues proof requests on start (47 ms)
13:20:28
✓ restores proof results on start (58 ms)
13:20:28
✓ only re-enqueues unfinished jobs (61 ms)
13:20:28
✓ saves job when enqueued (33 ms)
13:20:28
✓ does not retain job if database fails to save (5 ms)
13:20:28
✓ saves job result (33 ms)
13:20:28
✓ saves result even if database fails to save (32 ms)
13:20:28
✓ saves job error (36 ms)
13:20:28
✓ saves job error even if database fails to save (30 ms)
13:20:28
✓ does not save job result if job is unknown (7 ms)
13:20:28
✓ does not save job error if job is unknown (3 ms)
13:20:28
✓ cleans up old jobs periodically (415 ms)
13:20:28
13:20:28
Test Suites: 1 passed, 1 total
13:20:28
Tests: 96 passed, 96 total
13:20:28
Snapshots: 0 total
13:20:28
Time: 13.512 s
13:20:28
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
13:20:28
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?