Parent Log:
http://ci.aztec-labs.com/943200330be9ae5b
Command: 61f8e44af0fc4001 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/e56baa7f24bac54baf9e2f22f6f33ae6fa8b8c0f
Env: REF_NAME=gh-readonly-queue/next/pr-14891-76ca48a2187e3506bb464eae574e49476c2876ca CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 13 19:22:29 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14891_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
19:22:33 [19:22:33.470]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:898b1c9ba1edc6e4 epochNumber=42
{"provingJobId":"1:BASE_PARITY:898b1c9ba1edc6e4"}
19:22:33 [19:22:33.482]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:cb08f4b3042d3245 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb08f4b3042d3245"}
19:22:33 [19:22:33.517]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.518]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aedd488c052dd135 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aedd488c052dd135"}
19:22:33 [19:22:33.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c16359fa5a38f0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c16359fa5a38f0b"}
19:22:33 [19:22:33.522]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.524]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4356f6d4e5079d38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4356f6d4e5079d38"}
19:22:33 [19:22:33.525]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4356f6d4e5079d38 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4356f6d4e5079d38"}
19:22:33 [19:22:33.527]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.528]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf8792ace0422e90 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf8792ace0422e90"}
19:22:33 [19:22:33.529]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cf8792ace0422e90 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:cf8792ace0422e90"}
19:22:33 [19:22:33.530]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cf8792ace0422e90 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cf8792ace0422e90"}
19:22:33 [19:22:33.531]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cf8792ace0422e90 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:cf8792ace0422e90"}
19:22:33 [19:22:33.532]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.532]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25f05de0beb670d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:25f05de0beb670d0"}
19:22:33 [19:22:33.560]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.562]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.562]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d469f27d470ef5fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:d469f27d470ef5fd"}
19:22:33 [19:22:33.563]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d469f27d470ef5fd
{"provingJobId":"1:BASE_PARITY:d469f27d470ef5fd"}
19:22:33 [19:22:33.563]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d469f27d470ef5fd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d469f27d470ef5fd"}
19:22:33 [19:22:33.564]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d469f27d470ef5fd type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:d469f27d470ef5fd"}
19:22:33 [19:22:33.572]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.573]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f9c8c217d2cc7e98 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f9c8c217d2cc7e98"}
19:22:33 [19:22:33.574]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f9c8c217d2cc7e98
{"provingJobId":"1:BASE_PARITY:f9c8c217d2cc7e98"}
19:22:33 [19:22:33.574]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f9c8c217d2cc7e98 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f9c8c217d2cc7e98"}
19:22:33 [19:22:33.576]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.577]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e21394ece4561d9b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e21394ece4561d9b"}
19:22:33 [19:22:33.577]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e21394ece4561d9b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e21394ece4561d9b"}
19:22:33 [19:22:33.577]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e21394ece4561d9b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e21394ece4561d9b"}
19:22:33 [19:22:33.579]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.579]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:352291da6bccaf71 epochNumber=1
{"provingJobId":"1:BASE_PARITY:352291da6bccaf71"}
19:22:33 [19:22:33.580]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:352291da6bccaf71 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:352291da6bccaf71"}
19:22:33 [19:22:33.580]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:352291da6bccaf71 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:352291da6bccaf71"}
19:22:33 [19:22:33.581]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:852f106dc631574d epochNumber=0
{"provingJobId":"1:BASE_PARITY:852f106dc631574d"}
19:22:33 [19:22:33.582]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:852f106dc631574d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:852f106dc631574d"}
19:22:33 [19:22:33.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:491e876cdf9802dc epochNumber=0
{"provingJobId":"1:BASE_PARITY:491e876cdf9802dc"}
19:22:33 [19:22:33.583]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:852f106dc631574d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:852f106dc631574d"}
19:22:33 [19:22:33.583]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f8eb11114fbe7f0 epochNumber=0
{"provingJobId":"1:BASE_PARITY:0f8eb11114fbe7f0"}
19:22:33 [19:22:33.584]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.585]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.586]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f25b36df1eae887f epochNumber=2
{"provingJobId":"1:BASE_PARITY:f25b36df1eae887f"}
19:22:33 [19:22:33.586]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f1da0bde83e1cf33 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f1da0bde83e1cf33"}
19:22:33 [19:22:33.587]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.588]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6df7526c6cd65106 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6df7526c6cd65106"}
19:22:33 [19:22:33.589]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.589]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a65a5dbee365f36b epochNumber=1
{"provingJobId":"1:BASE_PARITY:a65a5dbee365f36b"}
19:22:33 [19:22:33.590]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c22938126db5e1d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c22938126db5e1d9"}
19:22:33 [19:22:33.590]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8f8b17dbde9a991 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e8f8b17dbde9a991"}
19:22:33 [19:22:33.590]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a91a862c7b7fb34b epochNumber=1
{"provingJobId":"1:BASE_PARITY:a91a862c7b7fb34b"}
19:22:33 [19:22:33.591]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.592]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a118cc7857409a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a118cc7857409a0"}
19:22:33 [19:22:33.592]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e383c394f1f57b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e383c394f1f57b9"}
19:22:33 [19:22:33.592]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b89df0a33b67ca43 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b89df0a33b67ca43"}
19:22:33 [19:22:33.593]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f176b166126a7836 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f176b166126a7836"}
19:22:33 [19:22:33.594]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0a35979f189ff0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c0a35979f189ff0e"}
19:22:33 [19:22:33.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a90b34ddab98cbf2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a90b34ddab98cbf2"}
19:22:33 [19:22:33.595]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dcd0f2f47a8202d4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dcd0f2f47a8202d4"}
19:22:33 [19:22:33.595]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:581d34aac65c33da epochNumber=1
{"provingJobId":"1:BASE_PARITY:581d34aac65c33da"}
19:22:33 [19:22:33.596]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.597]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:425284591fd099b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:425284591fd099b5"}
19:22:33 [19:22:33.597]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c076285b7542e6f epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c076285b7542e6f"}
19:22:33 [19:22:33.598]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:425284591fd099b5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:425284591fd099b5"}
19:22:33 [19:22:33.600]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.601]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:49c711a3bbd873e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:49c711a3bbd873e9"}
19:22:33 [19:22:33.601]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a385c782aa2f1fa7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a385c782aa2f1fa7"}
19:22:33 [19:22:33.602]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:49c711a3bbd873e9 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:49c711a3bbd873e9"}
19:22:33 [19:22:33.603]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.603]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c9140df95a91b5a epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c9140df95a91b5a"}
19:22:33 [19:22:33.603]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c0ba55b7da0f923 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c0ba55b7da0f923"}
19:22:33 [19:22:33.604]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3c9140df95a91b5a type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:3c9140df95a91b5a"}
19:22:33 [19:22:33.605]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.605]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be252e528641b318 epochNumber=1
{"provingJobId":"1:BASE_PARITY:be252e528641b318"}
19:22:33 [19:22:33.606]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:be252e528641b318
{"provingJobId":"1:BASE_PARITY:be252e528641b318"}
19:22:33 [19:22:33.606]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:be252e528641b318 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:be252e528641b318"}
19:22:33 [19:22:33.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3aeedc8f0c689515 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3aeedc8f0c689515"}
19:22:33 [19:22:33.607]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:be252e528641b318 has already been completed
{"provingJobId":"1:BASE_PARITY:be252e528641b318"}
19:22:33 [19:22:33.608]
INFO:
prover-client:proving-broker Proving Broker started
19:22:33 [19:22:33.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:042142e96c4833be epochNumber=1
{"provingJobId":"1:BASE_PARITY:042142e96c4833be"}
19:22:33 [19:22:33.609]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68e2db726ffc433f epochNumber=2
{"provingJobId":"1:BASE_PARITY:68e2db726ffc433f"}
19:22:34 [19:22:34.661]
INFO:
prover-client:proving-broker Proving Broker started
19:22:34 [19:22:34.666]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:042142e96c4833be settled=false
{"provingJobId":"1:BASE_PARITY:042142e96c4833be","status":"pending"}
19:22:34 [19:22:34.667]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:68e2db726ffc433f settled=false
{"provingJobId":"1:BASE_PARITY:68e2db726ffc433f","status":"pending"}
19:22:34 [19:22:34.668]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:042142e96c4833be type=BASE_PARITY startedAt=1749842553609 older agent has taken job
{"provingJobId":"1:BASE_PARITY:042142e96c4833be"}
19:22:34 [19:22:34.668]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:042142e96c4833be type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:042142e96c4833be"}
19:22:34 [19:22:34.670]
INFO:
prover-client:proving-broker Proving Broker started
19:22:34 [19:22:34.671]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e075a4ff280a7a11 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e075a4ff280a7a11"}
19:22:34 [19:22:34.671]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c17377021629c03d epochNumber=2
{"provingJobId":"1:BASE_PARITY:c17377021629c03d"}
19:22:35 [19:22:35.674]
INFO:
prover-client:proving-broker Proving Broker started
19:22:35 [19:22:35.675]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e075a4ff280a7a11 settled=false
{"provingJobId":"1:BASE_PARITY:e075a4ff280a7a11","status":"pending"}
19:22:35 [19:22:35.679]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c17377021629c03d settled=false
{"provingJobId":"1:BASE_PARITY:c17377021629c03d","status":"pending"}
19:22:35 [19:22:35.680]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e075a4ff280a7a11 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:e075a4ff280a7a11"}
19:22:35 [19:22:35.683]
INFO:
prover-client:proving-broker Proving Broker started
19:22:35 [19:22:35.684]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21a3238fc409e651 epochNumber=1
{"provingJobId":"1:BASE_PARITY:21a3238fc409e651"}
19:22:35 [19:22:35.685]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbc4775cc6b1c6e8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dbc4775cc6b1c6e8"}
19:22:36 [19:22:36.688]
INFO:
prover-client:proving-broker Proving Broker started
19:22:36 [19:22:36.688]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:21a3238fc409e651 settled=false
{"provingJobId":"1:BASE_PARITY:21a3238fc409e651","status":"pending"}
19:22:36 [19:22:36.689]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dbc4775cc6b1c6e8 settled=false
{"provingJobId":"1:BASE_PARITY:dbc4775cc6b1c6e8","status":"pending"}
19:22:36 [19:22:36.689]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:21a3238fc409e651 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:21a3238fc409e651"}
19:22:36 [19:22:36.690]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:21a3238fc409e651 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:21a3238fc409e651"}
19:22:36 [19:22:36.691]
INFO:
prover-client:proving-broker Proving Broker started
19:22:36 [19:22:36.691]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f46969f1e39ba36b epochNumber=1
{"provingJobId":"1:BASE_PARITY:f46969f1e39ba36b"}
19:22:36 [19:22:36.692]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50b00407dcebbf42 epochNumber=2
{"provingJobId":"1:BASE_PARITY:50b00407dcebbf42"}
19:22:36 [19:22:36.692]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f46969f1e39ba36b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f46969f1e39ba36b"}
19:22:36 [19:22:36.693]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:50b00407dcebbf42 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:50b00407dcebbf42"}
19:22:36 [19:22:36.694]
INFO:
prover-client:proving-broker Proving Broker started
19:22:36 [19:22:36.694]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:052d2829ec49aec6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:052d2829ec49aec6"}
19:22:36 [19:22:36.694]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90001e18ecd44422 epochNumber=2
{"provingJobId":"1:BASE_PARITY:90001e18ecd44422"}
19:22:36 [19:22:36.695]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:052d2829ec49aec6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:052d2829ec49aec6"}
19:22:36 [19:22:36.695]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:052d2829ec49aec6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:052d2829ec49aec6"}
19:22:36 [19:22:36.695]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:90001e18ecd44422 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:90001e18ecd44422"}
19:22:36 [19:22:36.696]
INFO:
prover-client:proving-broker Proving Broker started
19:22:36 [19:22:36.697]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ffcbe65d9882a1f6 err=test error
{"provingJoId":"1:BASE_PARITY:ffcbe65d9882a1f6"}
19:22:36 [19:22:36.697]
INFO:
prover-client:proving-broker Proving Broker started
19:22:36 [19:22:36.698]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:76dda36fa2d3ebee not found
{"provingJobId":"1:BASE_PARITY:76dda36fa2d3ebee"}
19:22:36 [19:22:36.699]
INFO:
prover-client:proving-broker Proving Broker started
19:22:36 [19:22:36.699]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc4f0e4f2d86ee68 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc4f0e4f2d86ee68"}
19:22:36 [19:22:36.700]
INFO:
prover-client:proving-broker Proving Broker started
19:22:36 [19:22:36.701]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3df94837d77dbcf9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3df94837d77dbcf9"}
19:22:36 [19:22:36.802]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3df94837d77dbcf9 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:3df94837d77dbcf9"}
19:22:36 [19:22:36.809]
INFO:
prover-client:proving-broker Proving Broker started
19:22:36 [19:22:36.810]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:048efadf99c7b909 epochNumber=1
{"provingJobId":"1:BASE_PARITY:048efadf99c7b909"}
19:22:36 [19:22:36.911]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:048efadf99c7b909 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:048efadf99c7b909"}
19:22:36 [19:22:36.917]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3bc18b89192c22e2 epochNumber=10
{"provingJobId":"1:BASE_PARITY:3bc18b89192c22e2"}
19:22:36 [19:22:36.937]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:37 [19:22:37.044]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.045]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40733a3418e7c3e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:40733a3418e7c3e3"}
19:22:37 [19:22:37.201]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:40733a3418e7c3e3 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:40733a3418e7c3e3"}
19:22:37 [19:22:37.250]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:befa8ee061184c7b epochNumber=1
{"provingJobId":"1:BASE_PARITY:befa8ee061184c7b"}
19:22:37 [19:22:37.251]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:befa8ee061184c7b type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:befa8ee061184c7b"}
19:22:37 [19:22:37.252]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.252]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6eff16ceba6df4a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6eff16ceba6df4a"}
19:22:37 [19:22:37.253]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a6eff16ceba6df4a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a6eff16ceba6df4a"}
19:22:37 [19:22:37.253]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a6eff16ceba6df4a type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:a6eff16ceba6df4a"}
19:22:37 [19:22:37.254]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5cca37bb36e37a4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:5cca37bb36e37a4b"}
19:22:37 [19:22:37.258]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5cca37bb36e37a4b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5cca37bb36e37a4b"}
19:22:37 [19:22:37.260]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.260]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58d115afc3733945 epochNumber=1
{"provingJobId":"1:BASE_PARITY:58d115afc3733945"}
19:22:37 [19:22:37.261]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:58d115afc3733945 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:58d115afc3733945"}
19:22:37 [19:22:37.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:293e595d4156ea1c epochNumber=3
{"provingJobId":"1:BASE_PARITY:293e595d4156ea1c"}
19:22:37 [19:22:37.284]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:37 [19:22:37.286]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:58d115afc3733945 err=test error
{"provingJoId":"1:BASE_PARITY:58d115afc3733945"}
19:22:37 [19:22:37.288]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.288]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5351658202ca5835 settled=false
{"provingJobId":"1:BASE_PARITY:5351658202ca5835","status":"pending"}
19:22:37 [19:22:37.288]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8cf0a6f7d170b9d4 settled=false
{"provingJobId":"1:BASE_PARITY:8cf0a6f7d170b9d4","status":"pending"}
19:22:37 [19:22:37.290]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.290]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:16e521eb8193cd0d settled=true
{"provingJobId":"1:BASE_PARITY:16e521eb8193cd0d","status":"fulfilled"}
19:22:37 [19:22:37.290]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:490a29fe4348ff70 settled=true
{"provingJobId":"2:BASE_PARITY:490a29fe4348ff70","status":"fulfilled"}
19:22:37 [19:22:37.291]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.292]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b55813a6ca5a0f0d settled=true
{"provingJobId":"1:BASE_PARITY:b55813a6ca5a0f0d","status":"fulfilled"}
19:22:37 [19:22:37.292]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:33c3bb2d7dba533c settled=false
{"provingJobId":"1:BASE_PARITY:33c3bb2d7dba533c","status":"pending"}
19:22:37 [19:22:37.293]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.294]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a860cb44e543426c epochNumber=1
{"provingJobId":"1:BASE_PARITY:a860cb44e543426c"}
19:22:37 [19:22:37.295]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.296]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:904891918c36169e epochNumber=1
{"provingJobId":"1:BASE_PARITY:904891918c36169e"}
19:22:37 [19:22:37.296]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:904891918c36169e: Error: db error: Error: db error
19:22:37 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:904891918c36169e"}
19:22:37 [19:22:37.297]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.298]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df71f4c4f730891e epochNumber=1
{"provingJobId":"1:BASE_PARITY:df71f4c4f730891e"}
19:22:37 [19:22:37.298]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:df71f4c4f730891e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:df71f4c4f730891e"}
19:22:37 [19:22:37.298]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:df71f4c4f730891e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:df71f4c4f730891e"}
19:22:37 [19:22:37.299]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.300]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca1d9f3a7a268af6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca1d9f3a7a268af6"}
19:22:37 [19:22:37.300]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ca1d9f3a7a268af6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ca1d9f3a7a268af6"}
19:22:37 [19:22:37.301]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ca1d9f3a7a268af6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ca1d9f3a7a268af6"}
19:22:37 [19:22:37.302]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:ca1d9f3a7a268af6: Error: db error
19:22:37 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:ca1d9f3a7a268af6"}
19:22:37 [19:22:37.303]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.304]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3db8ce74bf284a5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:3db8ce74bf284a5e"}
19:22:37 [19:22:37.304]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3db8ce74bf284a5e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3db8ce74bf284a5e"}
19:22:37 [19:22:37.304]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3db8ce74bf284a5e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3db8ce74bf284a5e"}
19:22:37 [19:22:37.305]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.306]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04c55ea2eec3a164 epochNumber=1
{"provingJobId":"1:BASE_PARITY:04c55ea2eec3a164"}
19:22:37 [19:22:37.306]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:04c55ea2eec3a164 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:04c55ea2eec3a164"}
19:22:37 [19:22:37.307]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:04c55ea2eec3a164 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:04c55ea2eec3a164"}
19:22:37 [19:22:37.308]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:04c55ea2eec3a164 jobErr=test error: Error: db error
19:22:37 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:04c55ea2eec3a164"}
19:22:37 [19:22:37.309]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.310]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fac3dd30dbcbd183 not found
{"provingJobId":"1:BASE_PARITY:fac3dd30dbcbd183"}
19:22:37 [19:22:37.311]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.312]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b4629ec4bbf113b7 err=test error
{"provingJoId":"1:BASE_PARITY:b4629ec4bbf113b7"}
19:22:37 [19:22:37.313]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.323]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:001eb35b88c438b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:001eb35b88c438b0"}
19:22:37 [19:22:37.324]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:001eb35b88c438b0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:001eb35b88c438b0"}
19:22:37 [19:22:37.324]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:001eb35b88c438b0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:001eb35b88c438b0"}
19:22:37 [19:22:37.335]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:2634c0653b3b835d epochNumber=2
{"provingJobId":"2:BASE_PARITY:2634c0653b3b835d"}
19:22:37 [19:22:37.335]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:2634c0653b3b835d type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:2634c0653b3b835d"}
19:22:37 [19:22:37.336]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2634c0653b3b835d type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:2634c0653b3b835d"}
19:22:37 [19:22:37.397]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:e95814bd9be222da epochNumber=3
{"provingJobId":"3:BASE_PARITY:e95814bd9be222da"}
19:22:37 [19:22:37.413]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:37 [19:22:37.459]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:65b2a402c429aade epochNumber=4
{"provingJobId":"4:BASE_PARITY:65b2a402c429aade"}
19:22:37 [19:22:37.467]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:37 [19:22:37.521]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:2796e07dfdab8508 epochNumber=5
{"provingJobId":"5:BASE_PARITY:2796e07dfdab8508"}
19:22:37 [19:22:37.543]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:37 [19:22:37.581]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.581]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a0d52726f527c1f epochNumber=42
{"provingJobId":"1:BASE_PARITY:1a0d52726f527c1f"}
19:22:37 [19:22:37.583]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-test5aVFrm/42 with map size 1073741824
19:22:37 [19:22:37.587]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.617]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:77eeea261fb82c1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:77eeea261fb82c1f"}
19:22:37 [19:22:37.622]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.623]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f079ce8a8f14f15 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f079ce8a8f14f15"}
19:22:37 [19:22:37.623]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBKUzaa/1 with map size 1073741824
19:22:37 [19:22:37.627]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.650]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a9b5ddeeea21cae epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a9b5ddeeea21cae"}
19:22:37 [19:22:37.668]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.669]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca5debe4a455dca4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca5debe4a455dca4"}
19:22:37 [19:22:37.675]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testI2av4U/1 with map size 1073741824
19:22:37 [19:22:37.682]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.710]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ca5debe4a455dca4 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ca5debe4a455dca4"}
19:22:37 [19:22:37.714]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.715]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6bd1b7e6fd3e9a0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:6bd1b7e6fd3e9a0e"}
19:22:37 [19:22:37.715]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzzwa4d/1 with map size 1073741824
19:22:37 [19:22:37.718]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.740]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6bd1b7e6fd3e9a0e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6bd1b7e6fd3e9a0e"}
19:22:37 [19:22:37.741]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6bd1b7e6fd3e9a0e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6bd1b7e6fd3e9a0e"}
19:22:37 [19:22:37.749]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6bd1b7e6fd3e9a0e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6bd1b7e6fd3e9a0e"}
19:22:37 [19:22:37.762]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.763]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:051104220e455f97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:051104220e455f97"}
19:22:37 [19:22:37.764]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuuQIqR/1 with map size 1073741824
19:22:37 [19:22:37.765]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.789]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.792]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.793]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ceebb92df94cc9a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ceebb92df94cc9a6"}
19:22:37 [19:22:37.793]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIq4Tn6/1 with map size 1073741824
19:22:37 [19:22:37.795]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.813]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ceebb92df94cc9a6
{"provingJobId":"1:BASE_PARITY:ceebb92df94cc9a6"}
19:22:37 [19:22:37.813]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ceebb92df94cc9a6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ceebb92df94cc9a6"}
19:22:37 [19:22:37.813]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ceebb92df94cc9a6 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ceebb92df94cc9a6"}
19:22:37 [19:22:37.822]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.822]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a2e386d7900415b epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a2e386d7900415b"}
19:22:37 [19:22:37.823]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuuXmcX/1 with map size 1073741824
19:22:37 [19:22:37.826]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.844]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3a2e386d7900415b
{"provingJobId":"1:BASE_PARITY:3a2e386d7900415b"}
19:22:37 [19:22:37.844]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3a2e386d7900415b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3a2e386d7900415b"}
19:22:37 [19:22:37.862]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.865]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6119661d008bead7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6119661d008bead7"}
19:22:37 [19:22:37.867]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWKnc3q/1 with map size 1073741824
19:22:37 [19:22:37.871]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.891]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6119661d008bead7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6119661d008bead7"}
19:22:37 [19:22:37.892]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6119661d008bead7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6119661d008bead7"}
19:22:37 [19:22:37.904]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.906]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed4a1429210aff93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed4a1429210aff93"}
19:22:37 [19:22:37.908]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzitBPB/1 with map size 1073741824
19:22:37 [19:22:37.912]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.943]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ed4a1429210aff93 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ed4a1429210aff93"}
19:22:37 [19:22:37.944]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ed4a1429210aff93 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ed4a1429210aff93"}
19:22:37 [19:22:37.967]
INFO:
prover-client:proving-broker Proving Broker started
19:22:37 [19:22:37.968]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2eb294d09783311d epochNumber=0
{"provingJobId":"1:BASE_PARITY:2eb294d09783311d"}
19:22:37 [19:22:37.973]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testU7Ulq5/0 with map size 1073741824
19:22:37 [19:22:37.976]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:37 [19:22:37.999]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2eb294d09783311d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2eb294d09783311d"}
19:22:37 [19:22:38.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:598531350b62ef8e epochNumber=0
{"provingJobId":"1:BASE_PARITY:598531350b62ef8e"}
19:22:37 [19:22:38.000]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2eb294d09783311d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2eb294d09783311d"}
19:22:37 [19:22:38.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4449d19bccd933b0 epochNumber=0
{"provingJobId":"1:BASE_PARITY:4449d19bccd933b0"}
19:22:38 [19:22:38.018]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.021]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.021]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0051048ef87794c5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0051048ef87794c5"}
19:22:38 [19:22:38.022]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcIYTvT/2 with map size 1073741824
19:22:38 [19:22:38.025]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.046]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf1191d6215bcbc7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf1191d6215bcbc7"}
19:22:38 [19:22:38.047]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcIYTvT/1 with map size 1073741824
19:22:38 [19:22:38.049]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.076]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47f341d53d93fd0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:47f341d53d93fd0b"}
19:22:38 [19:22:38.078]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQQ5XeT/1 with map size 1073741824
19:22:38 [19:22:38.082]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.103]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.103]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ba0bcab161cb354 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ba0bcab161cb354"}
19:22:38 [19:22:38.106]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testj7osyO/1 with map size 1073741824
19:22:38 [19:22:38.110]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.130]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb8bf2163709f4bf epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb8bf2163709f4bf"}
19:22:38 [19:22:38.138]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b31f9b8ab08c1d5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6b31f9b8ab08c1d5"}
19:22:38 [19:22:38.139]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testj7osyO/2 with map size 1073741824
19:22:38 [19:22:38.142]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6dba6334e058f078 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6dba6334e058f078"}
19:22:38 [19:22:38.180]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.181]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d1714361571ee7b epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d1714361571ee7b"}
19:22:38 [19:22:38.182]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUmCyNS/1 with map size 1073741824
19:22:38 [19:22:38.184]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.202]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b602d1d8c4a90123 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b602d1d8c4a90123"}
19:22:38 [19:22:38.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19b4dcc435905658 epochNumber=2
{"provingJobId":"1:BASE_PARITY:19b4dcc435905658"}
19:22:38 [19:22:38.209]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUmCyNS/2 with map size 1073741824
19:22:38 [19:22:38.212]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.229]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12278d742c2291fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:12278d742c2291fb"}
19:22:38 [19:22:38.240]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.240]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af7e6facf8ae0cf4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af7e6facf8ae0cf4"}
19:22:38 [19:22:38.241]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9gbbnt/1 with map size 1073741824
19:22:38 [19:22:38.243]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.276]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da5885b848a57a51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:da5885b848a57a51"}
19:22:38 [19:22:38.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02ea837209832479 epochNumber=2
{"provingJobId":"1:BASE_PARITY:02ea837209832479"}
19:22:38 [19:22:38.282]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test9gbbnt/2 with map size 1073741824
19:22:38 [19:22:38.284]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.300]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94378e5519c38fc9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:94378e5519c38fc9"}
19:22:38 [19:22:38.310]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.311]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17e8cbd2ec2eec09 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17e8cbd2ec2eec09"}
19:22:38 [19:22:38.312]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6rGFdu/1 with map size 1073741824
19:22:38 [19:22:38.313]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.333]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b5cd758a148f376 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b5cd758a148f376"}
19:22:38 [19:22:38.340]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:17e8cbd2ec2eec09 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:17e8cbd2ec2eec09"}
19:22:38 [19:22:38.349]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.350]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:584fa3932a87aaa0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:584fa3932a87aaa0"}
19:22:38 [19:22:38.351]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testG0BPGf/1 with map size 1073741824
19:22:38 [19:22:38.367]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.389]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b72b98937642856f epochNumber=1
{"provingJobId":"1:BASE_PARITY:b72b98937642856f"}
19:22:38 [19:22:38.395]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:584fa3932a87aaa0 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:584fa3932a87aaa0"}
19:22:38 [19:22:38.405]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.405]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2588f9b68072efa epochNumber=1
{"provingJobId":"1:BASE_PARITY:d2588f9b68072efa"}
19:22:38 [19:22:38.406]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxXDRzr/1 with map size 1073741824
19:22:38 [19:22:38.407]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.427]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bfbb6b21d67c7784 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bfbb6b21d67c7784"}
19:22:38 [19:22:38.434]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d2588f9b68072efa type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:d2588f9b68072efa"}
19:22:38 [19:22:38.439]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.440]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45c940a04839e8b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:45c940a04839e8b9"}
19:22:38 [19:22:38.441]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGbjhWY/1 with map size 1073741824
19:22:38 [19:22:38.443]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.465]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:45c940a04839e8b9
{"provingJobId":"1:BASE_PARITY:45c940a04839e8b9"}
19:22:38 [19:22:38.466]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:45c940a04839e8b9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:45c940a04839e8b9"}
19:22:38 [19:22:38.473]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1310bf1182f71669 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1310bf1182f71669"}
19:22:38 [19:22:38.481]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:45c940a04839e8b9 has already been completed
{"provingJobId":"1:BASE_PARITY:45c940a04839e8b9"}
19:22:38 [19:22:38.486]
INFO:
prover-client:proving-broker Proving Broker started
19:22:38 [19:22:38.486]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:209fd7195dd268ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:209fd7195dd268ec"}
19:22:38 [19:22:38.487]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3QOlxb/1 with map size 1073741824
19:22:38 [19:22:38.490]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:38 [19:22:38.508]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:499be6fd859553ca epochNumber=2
{"provingJobId":"1:BASE_PARITY:499be6fd859553ca"}
19:22:38 [19:22:38.509]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test3QOlxb/2 with map size 1073741824
19:22:38 [19:22:38.512]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:39 [19:22:39.582]
INFO:
prover-client:proving-broker Proving Broker started
19:22:39 [19:22:39.589]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:209fd7195dd268ec settled=false
{"provingJobId":"1:BASE_PARITY:209fd7195dd268ec","status":"pending"}
19:22:39 [19:22:39.590]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:499be6fd859553ca settled=false
{"provingJobId":"1:BASE_PARITY:499be6fd859553ca","status":"pending"}
19:22:39 [19:22:39.592]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:209fd7195dd268ec type=BASE_PARITY startedAt=1749842558530 older agent has taken job
{"provingJobId":"1:BASE_PARITY:209fd7195dd268ec"}
19:22:39 [19:22:39.592]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:209fd7195dd268ec type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:209fd7195dd268ec"}
19:22:39 [19:22:39.598]
INFO:
prover-client:proving-broker Proving Broker started
19:22:39 [19:22:39.599]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:921b98332c7535f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:921b98332c7535f8"}
19:22:39 [19:22:39.600]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFNKEQX/1 with map size 1073741824
19:22:39 [19:22:39.601]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:39 [19:22:39.624]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:377fcf4abbf9e1b1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:377fcf4abbf9e1b1"}
19:22:39 [19:22:39.625]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFNKEQX/2 with map size 1073741824
19:22:39 [19:22:39.628]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:40 [19:22:40.647]
INFO:
prover-client:proving-broker Proving Broker started
19:22:40 [19:22:40.650]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:921b98332c7535f8 settled=false
{"provingJobId":"1:BASE_PARITY:921b98332c7535f8","status":"pending"}
19:22:40 [19:22:40.652]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:377fcf4abbf9e1b1 settled=false
{"provingJobId":"1:BASE_PARITY:377fcf4abbf9e1b1","status":"pending"}
19:22:40 [19:22:40.653]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:921b98332c7535f8 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:921b98332c7535f8"}
19:22:40 [19:22:40.661]
INFO:
prover-client:proving-broker Proving Broker started
19:22:40 [19:22:40.663]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e666131fe173be7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e666131fe173be7"}
19:22:40 [19:22:40.665]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOBn3th/1 with map size 1073741824
19:22:40 [19:22:40.673]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:40 [19:22:40.694]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:607476f2e1a70313 epochNumber=2
{"provingJobId":"1:BASE_PARITY:607476f2e1a70313"}
19:22:40 [19:22:40.695]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOBn3th/2 with map size 1073741824
19:22:40 [19:22:40.701]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:41 [19:22:41.725]
INFO:
prover-client:proving-broker Proving Broker started
19:22:41 [19:22:41.728]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4e666131fe173be7 settled=false
{"provingJobId":"1:BASE_PARITY:4e666131fe173be7","status":"pending"}
19:22:41 [19:22:41.731]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:607476f2e1a70313 settled=false
{"provingJobId":"1:BASE_PARITY:607476f2e1a70313","status":"pending"}
19:22:41 [19:22:41.732]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4e666131fe173be7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4e666131fe173be7"}
19:22:41 [19:22:41.732]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4e666131fe173be7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4e666131fe173be7"}
19:22:41 [19:22:41.743]
INFO:
prover-client:proving-broker Proving Broker started
19:22:41 [19:22:41.744]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2a433faf7da4195 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2a433faf7da4195"}
19:22:41 [19:22:41.745]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUp8XXL/1 with map size 1073741824
19:22:41 [19:22:41.749]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:41 [19:22:41.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9578ef9da3ecbdd0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9578ef9da3ecbdd0"}
19:22:41 [19:22:41.777]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUp8XXL/2 with map size 1073741824
19:22:41 [19:22:41.780]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:41 [19:22:41.802]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f2a433faf7da4195 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f2a433faf7da4195"}
19:22:41 [19:22:41.809]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9578ef9da3ecbdd0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9578ef9da3ecbdd0"}
19:22:41 [19:22:41.821]
INFO:
prover-client:proving-broker Proving Broker started
19:22:41 [19:22:41.822]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd04475c828dea54 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd04475c828dea54"}
19:22:41 [19:22:41.824]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testptSA4Z/1 with map size 1073741824
19:22:41 [19:22:41.827]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:41 [19:22:41.848]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b780d77bf5a333b epochNumber=2
{"provingJobId":"1:BASE_PARITY:9b780d77bf5a333b"}
19:22:41 [19:22:41.849]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testptSA4Z/2 with map size 1073741824
19:22:41 [19:22:41.851]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:41 [19:22:41.879]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bd04475c828dea54 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bd04475c828dea54"}
19:22:41 [19:22:41.879]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bd04475c828dea54 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bd04475c828dea54"}
19:22:41 [19:22:41.887]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9b780d77bf5a333b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9b780d77bf5a333b"}
19:22:41 [19:22:41.901]
INFO:
prover-client:proving-broker Proving Broker started
19:22:41 [19:22:41.902]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d475d05731bb3220 err=test error
{"provingJoId":"1:BASE_PARITY:d475d05731bb3220"}
19:22:41 [19:22:41.907]
INFO:
prover-client:proving-broker Proving Broker started
19:22:41 [19:22:41.909]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:610654e417be0136 not found
{"provingJobId":"1:BASE_PARITY:610654e417be0136"}
19:22:41 [19:22:41.912]
INFO:
prover-client:proving-broker Proving Broker started
19:22:41 [19:22:41.913]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4409b2cff1618a6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:4409b2cff1618a6a"}
19:22:41 [19:22:41.914]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRHd2d9/1 with map size 1073741824
19:22:41 [19:22:41.916]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:41 [19:22:41.944]
INFO:
prover-client:proving-broker Proving Broker started
19:22:41 [19:22:41.945]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5d49f64baa69d3c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f5d49f64baa69d3c"}
19:22:41 [19:22:41.946]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3oHzjn/1 with map size 1073741824
19:22:41 [19:22:41.951]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.097]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f5d49f64baa69d3c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f5d49f64baa69d3c"}
19:22:42 [19:22:42.103]
INFO:
prover-client:proving-broker Proving Broker started
19:22:42 [19:22:42.104]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:228064c21136743c epochNumber=1
{"provingJobId":"1:BASE_PARITY:228064c21136743c"}
19:22:42 [19:22:42.105]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt3b0nJ/1 with map size 1073741824
19:22:42 [19:22:42.107]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.231]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:228064c21136743c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:228064c21136743c"}
19:22:42 [19:22:42.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c97c8c49178ecc84 epochNumber=10
{"provingJobId":"1:BASE_PARITY:c97c8c49178ecc84"}
19:22:42 [19:22:42.233]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testt3b0nJ/10 with map size 1073741824
19:22:42 [19:22:42.235]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.257]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:42 [19:22:42.257]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:22:42 [19:22:42.383]
INFO:
prover-client:proving-broker Proving Broker started
19:22:42 [19:22:42.384]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e5bfafb9765b4a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e5bfafb9765b4a1"}
19:22:42 [19:22:42.385]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVJWSxF/1 with map size 1073741824
19:22:42 [19:22:42.387]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.561]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3e5bfafb9765b4a1 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:3e5bfafb9765b4a1"}
19:22:42 [19:22:42.607]
INFO:
prover-client:proving-broker Proving Broker started
19:22:42 [19:22:42.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57d8430ce5d9c087 epochNumber=1
{"provingJobId":"1:BASE_PARITY:57d8430ce5d9c087"}
19:22:42 [19:22:42.609]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test19FjmF/1 with map size 1073741824
19:22:42 [19:22:42.611]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.632]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:57d8430ce5d9c087 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:57d8430ce5d9c087"}
19:22:42 [19:22:42.636]
INFO:
prover-client:proving-broker Proving Broker started
19:22:42 [19:22:42.646]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44455fb5caf2011d epochNumber=1
{"provingJobId":"1:BASE_PARITY:44455fb5caf2011d"}
19:22:42 [19:22:42.651]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTBT5Y1/1 with map size 1073741824
19:22:42 [19:22:42.658]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.682]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:44455fb5caf2011d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:44455fb5caf2011d"}
19:22:42 [19:22:42.683]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:44455fb5caf2011d type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:44455fb5caf2011d"}
19:22:42 [19:22:42.699]
INFO:
prover-client:proving-broker Proving Broker started
19:22:42 [19:22:42.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:243741208fa2d03f epochNumber=1
{"provingJobId":"1:BASE_PARITY:243741208fa2d03f"}
19:22:42 [19:22:42.701]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIUiTns/1 with map size 1073741824
19:22:42 [19:22:42.704]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.723]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:243741208fa2d03f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:243741208fa2d03f"}
19:22:42 [19:22:42.735]
INFO:
prover-client:proving-broker Proving Broker started
19:22:42 [19:22:42.736]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:696622d270c2ace3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:696622d270c2ace3"}
19:22:42 [19:22:42.738]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoyU0ic/1 with map size 1073741824
19:22:42 [19:22:42.742]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.760]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:696622d270c2ace3 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:696622d270c2ace3"}
19:22:42 [19:22:42.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:49718f9d3ca0cee1 epochNumber=3
{"provingJobId":"1:BASE_PARITY:49718f9d3ca0cee1"}
19:22:42 [19:22:42.770]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testoyU0ic/3 with map size 1073741824
19:22:42 [19:22:42.773]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.820]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:42 [19:22:42.820]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:22:42 [19:22:42.822]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:696622d270c2ace3 err=test error
{"provingJoId":"1:BASE_PARITY:696622d270c2ace3"}
19:22:42 [19:22:42.830]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcrzMqY/1 with map size 1073741824
19:22:42 [19:22:42.838]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.862]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcrzMqY/2 with map size 1073741824
19:22:42 [19:22:42.866]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.889]
INFO:
prover-client:proving-broker Proving Broker started
19:22:42 [19:22:42.893]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2c2b3395a320c749 settled=false
{"provingJobId":"1:BASE_PARITY:2c2b3395a320c749","status":"pending"}
19:22:42 [19:22:42.896]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b9b8a0ef02bce5d5 settled=false
{"provingJobId":"1:BASE_PARITY:b9b8a0ef02bce5d5","status":"pending"}
19:22:42 [19:22:42.912]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlKjrNA/1 with map size 1073741824
19:22:42 [19:22:42.916]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.941]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlKjrNA/2 with map size 1073741824
19:22:42 [19:22:42.943]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:42 [19:22:42.980]
INFO:
prover-client:proving-broker Proving Broker started
19:22:42 [19:22:42.983]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e8ff6f63d8f10e53 settled=true
{"provingJobId":"1:BASE_PARITY:e8ff6f63d8f10e53","status":"fulfilled"}
19:22:42 [19:22:42.985]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:27b10389e4f640bf settled=true
{"provingJobId":"2:BASE_PARITY:27b10389e4f640bf","status":"fulfilled"}
19:22:43 [19:22:43.029]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFySDkX/1 with map size 1073741824
19:22:43 [19:22:43.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.058]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFySDkX/2 with map size 1073741824
19:22:43 [19:22:43.060]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.082]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.083]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3717caf2f5f4f4a1 settled=true
{"provingJobId":"1:BASE_PARITY:3717caf2f5f4f4a1","status":"fulfilled"}
19:22:43 [19:22:43.085]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cd9486aac85bfdf2 settled=false
{"provingJobId":"1:BASE_PARITY:cd9486aac85bfdf2","status":"pending"}
19:22:43 [19:22:43.090]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.091]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d1a2132bfd445e4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d1a2132bfd445e4"}
19:22:43 [19:22:43.091]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test03Tkn4/1 with map size 1073741824
19:22:43 [19:22:43.092]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.119]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:488fd6aa2c6dbe12 epochNumber=1
{"provingJobId":"1:BASE_PARITY:488fd6aa2c6dbe12"}
19:22:43 [19:22:43.120]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:488fd6aa2c6dbe12: Error: db error: Error: db error
19:22:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:488fd6aa2c6dbe12"}
19:22:43 [19:22:43.126]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ceda0fa586a6f234 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ceda0fa586a6f234"}
19:22:43 [19:22:43.127]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testymo2Wy/1 with map size 1073741824
19:22:43 [19:22:43.130]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.153]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ceda0fa586a6f234 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ceda0fa586a6f234"}
19:22:43 [19:22:43.153]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ceda0fa586a6f234 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ceda0fa586a6f234"}
19:22:43 [19:22:43.166]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.167]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:522c57641b0974e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:522c57641b0974e2"}
19:22:43 [19:22:43.168]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvAzajh/1 with map size 1073741824
19:22:43 [19:22:43.170]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.193]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:522c57641b0974e2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:522c57641b0974e2"}
19:22:43 [19:22:43.193]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:522c57641b0974e2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:522c57641b0974e2"}
19:22:43 [19:22:43.194]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:522c57641b0974e2: Error: db error
19:22:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:522c57641b0974e2"}
19:22:43 [19:22:43.199]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:400c8f6bf16e58ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:400c8f6bf16e58ad"}
19:22:43 [19:22:43.201]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJMBVR2/1 with map size 1073741824
19:22:43 [19:22:43.204]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.222]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:400c8f6bf16e58ad type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:400c8f6bf16e58ad"}
19:22:43 [19:22:43.223]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:400c8f6bf16e58ad type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:400c8f6bf16e58ad"}
19:22:43 [19:22:43.232]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.233]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec583f2bfefb6649 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec583f2bfefb6649"}
19:22:43 [19:22:43.234]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testErp5RG/1 with map size 1073741824
19:22:43 [19:22:43.238]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.258]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ec583f2bfefb6649 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ec583f2bfefb6649"}
19:22:43 [19:22:43.259]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ec583f2bfefb6649 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ec583f2bfefb6649"}
19:22:43 [19:22:43.260]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:ec583f2bfefb6649 jobErr=test error: Error: db error
19:22:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:ec583f2bfefb6649"}
19:22:43 [19:22:43.263]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.264]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:066de6d83a45c82e not found
{"provingJobId":"1:BASE_PARITY:066de6d83a45c82e"}
19:22:43 [19:22:43.268]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.269]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d25aafcfe546f4e4 err=test error
{"provingJoId":"1:BASE_PARITY:d25aafcfe546f4e4"}
19:22:43 [19:22:43.273]
INFO:
prover-client:proving-broker Proving Broker started
19:22:43 [19:22:43.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4aa315e4adc25c2e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4aa315e4adc25c2e"}
19:22:43 [19:22:43.285]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJX5tbX/1 with map size 1073741824
19:22:43 [19:22:43.289]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.309]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4aa315e4adc25c2e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4aa315e4adc25c2e"}
19:22:43 [19:22:43.310]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4aa315e4adc25c2e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4aa315e4adc25c2e"}
19:22:43 [19:22:43.325]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:b408c8570279c52f epochNumber=2
{"provingJobId":"2:BASE_PARITY:b408c8570279c52f"}
19:22:43 [19:22:43.326]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJX5tbX/2 with map size 1073741824
19:22:43 [19:22:43.329]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.347]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:b408c8570279c52f type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:b408c8570279c52f"}
19:22:43 [19:22:43.348]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:b408c8570279c52f type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:b408c8570279c52f"}
19:22:43 [19:22:43.414]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:9213318b221ec0ae epochNumber=3
{"provingJobId":"3:BASE_PARITY:9213318b221ec0ae"}
19:22:43 [19:22:43.415]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testJX5tbX/3 with map size 1073741824
19:22:43 [19:22:43.418]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.450]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:43 [19:22:43.451]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:22:43 [19:22:43.498]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:3b49bef54ebd60bd epochNumber=4
{"provingJobId":"4:BASE_PARITY:3b49bef54ebd60bd"}
19:22:43 [19:22:43.500]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testJX5tbX/4 with map size 1073741824
19:22:43 [19:22:43.502]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.527]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:43 [19:22:43.528]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
19:22:43 [19:22:43.579]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:5efff02ef9cb169e epochNumber=5
{"provingJobId":"5:BASE_PARITY:5efff02ef9cb169e"}
19:22:43 [19:22:43.580]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testJX5tbX/5 with map size 1073741824
19:22:43 [19:22:43.582]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:22:43 [19:22:43.604]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:22:43 [19:22:43.605]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
19:22:43
PASS src/proving_broker/proving_broker.test.ts (
13.19 s)
19:22:43 ProvingBroker
19:22:43 Producer API
19:22:43
✓ refuses stale jobs (49 ms)
19:22:43
✓ enqueues jobs (5 ms)
19:22:43
✓ ignores duplicate jobs (4 ms)
19:22:43
✓ reports correct status when enqueuing repeat jobs (4 ms)
19:22:43
✓ throws an error in case of duplicate job IDs (28 ms)
19:22:43
✓ returns not-found status for non-existing jobs (2 ms)
19:22:43
✓ cancels jobs in queue (10 ms)
19:22:43
✓ cancels jobs in-progress (3 ms)
19:22:43
✓ returns job result if successful (3 ms)
19:22:43
✓ returns job error if failed (3 ms)
19:22:43
✓ correctly returns job status for concurrent writes (3 ms)
19:22:43
✓ refuses stale jobs (44 ms)
19:22:43
✓ enqueues jobs (46 ms)
19:22:43
✓ ignores duplicate jobs (46 ms)
19:22:43
✓ reports correct status when enqueuing repeat jobs (41 ms)
19:22:43
✓ throws an error in case of duplicate job IDs (34 ms)
19:22:43
✓ returns not-found status for non-existing jobs (3 ms)
19:22:43
✓ cancels jobs in queue (29 ms)
19:22:43
✓ cancels jobs in-progress (31 ms)
19:22:43
✓ returns job result if successful (46 ms)
19:22:43
✓ returns job error if failed (58 ms)
19:22:43
✓ correctly returns job status for concurrent writes (56 ms)
19:22:43 Consumer API
19:22:43
✓ returns undefined if no jobs are available (1 ms)
19:22:43
✓ returns jobs in priority order (2 ms)
19:22:43
✓ returns undefined if no jobs are available for the given allowList (1 ms)
19:22:43
✓ returns a job if it is in the allowList (2 ms)
19:22:43
✓ returns the most important job if it is in the allowList (2 ms)
19:22:43
✓ returns any job if filter is empty (3 ms)
19:22:43
✓ returns a new job when reporting job success (3 ms)
19:22:43
✓ returns a new job when reporting permanent error (2 ms)
19:22:43
✓ returns a new job when reporting retry-able error (2 ms)
19:22:43
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
19:22:43
✓ returns a new job if job is already in progress elsewhere (1062 ms)
19:22:43
✓ avoids sending the same job to a new agent after a restart (1012 ms)
19:22:43
✓ avoids sending a completed job to a new agent after a restart (1007 ms)
19:22:43
✓ tracks job result if in progress (2 ms)
19:22:43
✓ tracks job result even if job is in queue (3 ms)
19:22:43
✓ ignores reported job error if unknown job (1 ms)
19:22:43
✓ ignores job result if unknown job (1 ms)
19:22:43
✓ returns undefined if no jobs are available (3 ms)
19:22:43
✓ returns jobs in priority order (55 ms)
19:22:43
✓ returns undefined if no jobs are available for the given allowList (26 ms)
19:22:43
✓ returns a job if it is in the allowList (77 ms)
19:22:43
✓ returns the most important job if it is in the allowList (59 ms)
19:22:43
✓ returns any job if filter is empty (70 ms)
19:22:43
✓ returns a new job when reporting job success (38 ms)
19:22:43
✓ returns a new job when reporting permanent error (54 ms)
19:22:43
✓ returns a new job when reporting retry-able error (34 ms)
19:22:43
✓ returns a new job when reporting progress if current one is cancelled (47 ms)
19:22:43
✓ returns a new job if job is already in progress elsewhere (1111 ms)
19:22:43
✓ avoids sending the same job to a new agent after a restart (1060 ms)
19:22:43
✓ avoids sending a completed job to a new agent after a restart (1085 ms)
19:22:43
✓ tracks job result if in progress (77 ms)
19:22:43
✓ tracks job result even if job is in queue (78 ms)
19:22:43
✓ ignores reported job error if unknown job (5 ms)
19:22:43
✓ ignores job result if unknown job (6 ms)
19:22:43 Timeouts
19:22:43
✓ tracks in progress jobs (1 ms)
19:22:43
✓ re-enqueues jobs that time out (108 ms)
19:22:43
✓ cancel stale jobs that time out (235 ms)
19:22:43
✓ keeps the jobs in progress while it is alive (204 ms)
19:22:43
✓ tracks in progress jobs (28 ms)
19:22:43
✓ re-enqueues jobs that time out (160 ms)
19:22:43
✓ cancel stale jobs that time out (282 ms)
19:22:43
✓ keeps the jobs in progress while it is alive (223 ms)
19:22:43 Retries
19:22:43
✓ retries jobs (2 ms)
19:22:43
✓ retries up to a maximum number of times (2 ms)
19:22:43
✓ passing retry=false does not retry (5 ms)
19:22:43
✓ does not retry if job is stale (28 ms)
19:22:43
✓ retries jobs (28 ms)
19:22:43
✓ retries up to a maximum number of times (60 ms)
19:22:43
✓ passing retry=false does not retry (36 ms)
19:22:43
✓ does not retry if job is stale (92 ms)
19:22:43 Database management
19:22:43
✓ re-enqueues proof requests on start (2 ms)
19:22:43
✓ restores proof results on start (1 ms)
19:22:43
✓ only re-enqueues unfinished jobs (1 ms)
19:22:43
✓ saves job when enqueued (2 ms)
19:22:43
✓ does not retain job if database fails to save (2 ms)
19:22:43
✓ saves job result (2 ms)
19:22:43
✓ saves result even if database fails to save (3 ms)
19:22:43
✓ saves job error (2 ms)
19:22:43
✓ saves job error even if database fails to save (4 ms)
19:22:43
✓ does not save job result if job is unknown (2 ms)
19:22:43
✓ does not save job error if job is unknown (1 ms)
19:22:43
✓ cleans up old jobs periodically (264 ms)
19:22:43
✓ re-enqueues proof requests on start (78 ms)
19:22:43
✓ restores proof results on start (82 ms)
19:22:43
✓ only re-enqueues unfinished jobs (100 ms)
19:22:43
✓ saves job when enqueued (29 ms)
19:22:43
✓ does not retain job if database fails to save (5 ms)
19:22:43
✓ saves job result (40 ms)
19:22:43
✓ saves result even if database fails to save (33 ms)
19:22:43
✓ saves job error (33 ms)
19:22:43
✓ saves job error even if database fails to save (32 ms)
19:22:43
✓ does not save job result if job is unknown (3 ms)
19:22:43
✓ does not save job error if job is unknown (5 ms)
19:22:43
✓ cleans up old jobs periodically (388 ms)
19:22:43
19:22:43
Test Suites: 1 passed, 1 total
19:22:43
Tests: 96 passed, 96 total
19:22:43
Snapshots: 0 total
19:22:43
Time: 13.284 s
19:22:43
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
19:22:43
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?