Parent Log:
http://ci.aztec-labs.com/c70586e10542d299
Command: c9b93c46acbfae2b yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/88c86d76e23c515deea689031fe891949d47ec49
Env: REF_NAME=gh-readonly-queue/next/pr-15152-3aafa23f28c401f1c88d60325230f1ce801d72a6 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Thu Jun 19 10:20:09 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15152_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
10:20:13 [10:20:13.573]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3397ceef616ea1e2 epochNumber=42
{"provingJobId":"1:BASE_PARITY:3397ceef616ea1e2"}
10:20:13 [10:20:13.586]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:547e8106dde8805d epochNumber=1
{"provingJobId":"1:BASE_PARITY:547e8106dde8805d"}
10:20:13 [10:20:13.629]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.630]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9011b64be5bf457 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c9011b64be5bf457"}
10:20:13 [10:20:13.632]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2c57ba56cd0963b epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2c57ba56cd0963b"}
10:20:13 [10:20:13.634]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.635]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0c767e313211ff2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0c767e313211ff2"}
10:20:13 [10:20:13.636]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b0c767e313211ff2 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b0c767e313211ff2"}
10:20:13 [10:20:13.638]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.639]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34418f1d6626cba0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:34418f1d6626cba0"}
10:20:13 [10:20:13.640]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:34418f1d6626cba0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:34418f1d6626cba0"}
10:20:13 [10:20:13.641]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:34418f1d6626cba0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:34418f1d6626cba0"}
10:20:13 [10:20:13.642]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:34418f1d6626cba0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:34418f1d6626cba0"}
10:20:13 [10:20:13.643]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.644]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b689bc3f458804d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b689bc3f458804d4"}
10:20:13 [10:20:13.674]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.676]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.677]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f13c4e85319589e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f13c4e85319589e7"}
10:20:13 [10:20:13.678]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f13c4e85319589e7
{"provingJobId":"1:BASE_PARITY:f13c4e85319589e7"}
10:20:13 [10:20:13.678]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f13c4e85319589e7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f13c4e85319589e7"}
10:20:13 [10:20:13.679]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f13c4e85319589e7 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f13c4e85319589e7"}
10:20:13 [10:20:13.681]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08c035c24dacc552 epochNumber=1
{"provingJobId":"1:BASE_PARITY:08c035c24dacc552"}
10:20:13 [10:20:13.684]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:08c035c24dacc552
{"provingJobId":"1:BASE_PARITY:08c035c24dacc552"}
10:20:13 [10:20:13.684]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:08c035c24dacc552 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:08c035c24dacc552"}
10:20:13 [10:20:13.686]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.687]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e96b63bf9e3c9f51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e96b63bf9e3c9f51"}
10:20:13 [10:20:13.688]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e96b63bf9e3c9f51 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e96b63bf9e3c9f51"}
10:20:13 [10:20:13.689]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e96b63bf9e3c9f51 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e96b63bf9e3c9f51"}
10:20:13 [10:20:13.693]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.695]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:baef541358bd9dcd epochNumber=1
{"provingJobId":"1:BASE_PARITY:baef541358bd9dcd"}
10:20:13 [10:20:13.696]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:baef541358bd9dcd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:baef541358bd9dcd"}
10:20:13 [10:20:13.697]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:baef541358bd9dcd type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:baef541358bd9dcd"}
10:20:13 [10:20:13.699]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.699]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbd19df2bd6ea22c epochNumber=0
{"provingJobId":"1:BASE_PARITY:dbd19df2bd6ea22c"}
10:20:13 [10:20:13.700]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dbd19df2bd6ea22c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dbd19df2bd6ea22c"}
10:20:13 [10:20:13.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1f8bc2020aa4547 epochNumber=0
{"provingJobId":"1:BASE_PARITY:a1f8bc2020aa4547"}
10:20:13 [10:20:13.701]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dbd19df2bd6ea22c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dbd19df2bd6ea22c"}
10:20:13 [10:20:13.702]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da101f035953ab2f epochNumber=0
{"provingJobId":"1:BASE_PARITY:da101f035953ab2f"}
10:20:13 [10:20:13.715]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.717]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.718]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7b007a37cd46daf epochNumber=2
{"provingJobId":"1:BASE_PARITY:e7b007a37cd46daf"}
10:20:13 [10:20:13.719]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77475a393402eb8d epochNumber=1
{"provingJobId":"1:BASE_PARITY:77475a393402eb8d"}
10:20:13 [10:20:13.721]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.722]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43f670f49deaab91 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43f670f49deaab91"}
10:20:13 [10:20:13.723]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.724]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10506f20de458dba epochNumber=1
{"provingJobId":"1:BASE_PARITY:10506f20de458dba"}
10:20:13 [10:20:13.724]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09037bc28390225d epochNumber=1
{"provingJobId":"1:BASE_PARITY:09037bc28390225d"}
10:20:13 [10:20:13.724]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c83833ad9214e187 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c83833ad9214e187"}
10:20:13 [10:20:13.725]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc71b6081085282f epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc71b6081085282f"}
10:20:13 [10:20:13.726]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.726]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d538718b7850f379 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d538718b7850f379"}
10:20:13 [10:20:13.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3e1c9703c5be376 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e3e1c9703c5be376"}
10:20:13 [10:20:13.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ccf49001af57c53 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4ccf49001af57c53"}
10:20:13 [10:20:13.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d142e4210e44631c epochNumber=1
{"provingJobId":"1:BASE_PARITY:d142e4210e44631c"}
10:20:13 [10:20:13.729]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.729]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9482b0a26edf17b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9482b0a26edf17b1"}
10:20:13 [10:20:13.729]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e17a7e56f576461 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e17a7e56f576461"}
10:20:13 [10:20:13.730]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a3f13d3945e3662 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3a3f13d3945e3662"}
10:20:13 [10:20:13.730]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94e288e5cd84a101 epochNumber=1
{"provingJobId":"1:BASE_PARITY:94e288e5cd84a101"}
10:20:13 [10:20:13.731]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.732]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:641ad7df99d4d8a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:641ad7df99d4d8a6"}
10:20:13 [10:20:13.732]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d83d6a91733743c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d83d6a91733743c4"}
10:20:13 [10:20:13.732]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:641ad7df99d4d8a6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:641ad7df99d4d8a6"}
10:20:13 [10:20:13.734]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.734]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:37f12056fae34ceb epochNumber=1
{"provingJobId":"1:BASE_PARITY:37f12056fae34ceb"}
10:20:13 [10:20:13.735]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0747e6202b3b55a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0747e6202b3b55a5"}
10:20:13 [10:20:13.735]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:37f12056fae34ceb type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:37f12056fae34ceb"}
10:20:13 [10:20:13.736]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.736]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b458890277d3b6cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:b458890277d3b6cd"}
10:20:13 [10:20:13.737]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd370b8b87e48866 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd370b8b87e48866"}
10:20:13 [10:20:13.737]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b458890277d3b6cd type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:b458890277d3b6cd"}
10:20:13 [10:20:13.738]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.739]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f1695f08599f223 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f1695f08599f223"}
10:20:13 [10:20:13.739]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9f1695f08599f223
{"provingJobId":"1:BASE_PARITY:9f1695f08599f223"}
10:20:13 [10:20:13.740]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9f1695f08599f223 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:9f1695f08599f223"}
10:20:13 [10:20:13.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d4ad16f432783ef epochNumber=1
{"provingJobId":"1:BASE_PARITY:5d4ad16f432783ef"}
10:20:13 [10:20:13.740]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9f1695f08599f223 has already been completed
{"provingJobId":"1:BASE_PARITY:9f1695f08599f223"}
10:20:13 [10:20:13.741]
INFO:
prover-client:proving-broker Proving Broker started
10:20:13 [10:20:13.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8fd7d10772bf7b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e8fd7d10772bf7b4"}
10:20:13 [10:20:13.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b954103be6ee4447 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b954103be6ee4447"}
10:20:14 [10:20:14.797]
INFO:
prover-client:proving-broker Proving Broker started
10:20:14 [10:20:14.798]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e8fd7d10772bf7b4 settled=false
{"provingJobId":"1:BASE_PARITY:e8fd7d10772bf7b4","status":"pending"}
10:20:14 [10:20:14.798]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b954103be6ee4447 settled=false
{"provingJobId":"1:BASE_PARITY:b954103be6ee4447","status":"pending"}
10:20:14 [10:20:14.799]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e8fd7d10772bf7b4 type=BASE_PARITY startedAt=1750328413743 older agent has taken job
{"provingJobId":"1:BASE_PARITY:e8fd7d10772bf7b4"}
10:20:14 [10:20:14.799]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e8fd7d10772bf7b4 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:e8fd7d10772bf7b4"}
10:20:14 [10:20:14.800]
INFO:
prover-client:proving-broker Proving Broker started
10:20:14 [10:20:14.801]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6eb14dd025f2042c epochNumber=1
{"provingJobId":"1:BASE_PARITY:6eb14dd025f2042c"}
10:20:14 [10:20:14.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1854a71db106b0d8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1854a71db106b0d8"}
10:20:15 [10:20:15.802]
INFO:
prover-client:proving-broker Proving Broker started
10:20:15 [10:20:15.802]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6eb14dd025f2042c settled=false
{"provingJobId":"1:BASE_PARITY:6eb14dd025f2042c","status":"pending"}
10:20:15 [10:20:15.803]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1854a71db106b0d8 settled=false
{"provingJobId":"1:BASE_PARITY:1854a71db106b0d8","status":"pending"}
10:20:15 [10:20:15.803]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6eb14dd025f2042c type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:6eb14dd025f2042c"}
10:20:15 [10:20:15.805]
INFO:
prover-client:proving-broker Proving Broker started
10:20:15 [10:20:15.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4254518554e7b439 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4254518554e7b439"}
10:20:15 [10:20:15.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:326e38f05a63d10a epochNumber=2
{"provingJobId":"1:BASE_PARITY:326e38f05a63d10a"}
10:20:16 [10:20:16.807]
INFO:
prover-client:proving-broker Proving Broker started
10:20:16 [10:20:16.807]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4254518554e7b439 settled=false
{"provingJobId":"1:BASE_PARITY:4254518554e7b439","status":"pending"}
10:20:16 [10:20:16.808]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:326e38f05a63d10a settled=false
{"provingJobId":"1:BASE_PARITY:326e38f05a63d10a","status":"pending"}
10:20:16 [10:20:16.808]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4254518554e7b439 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4254518554e7b439"}
10:20:16 [10:20:16.809]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4254518554e7b439 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4254518554e7b439"}
10:20:16 [10:20:16.810]
INFO:
prover-client:proving-broker Proving Broker started
10:20:16 [10:20:16.811]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:baae3bd03c215faa epochNumber=1
{"provingJobId":"1:BASE_PARITY:baae3bd03c215faa"}
10:20:16 [10:20:16.811]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19d4f16fa2d09553 epochNumber=2
{"provingJobId":"1:BASE_PARITY:19d4f16fa2d09553"}
10:20:16 [10:20:16.811]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:baae3bd03c215faa type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:baae3bd03c215faa"}
10:20:16 [10:20:16.812]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:19d4f16fa2d09553 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:19d4f16fa2d09553"}
10:20:16 [10:20:16.813]
INFO:
prover-client:proving-broker Proving Broker started
10:20:16 [10:20:16.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ccd2d2319faad496 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ccd2d2319faad496"}
10:20:16 [10:20:16.814]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80ce4191feea4b14 epochNumber=2
{"provingJobId":"1:BASE_PARITY:80ce4191feea4b14"}
10:20:16 [10:20:16.814]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ccd2d2319faad496 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ccd2d2319faad496"}
10:20:16 [10:20:16.814]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ccd2d2319faad496 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ccd2d2319faad496"}
10:20:16 [10:20:16.815]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:80ce4191feea4b14 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:80ce4191feea4b14"}
10:20:16 [10:20:16.816]
INFO:
prover-client:proving-broker Proving Broker started
10:20:16 [10:20:16.816]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0e449421da62e439 err=test error
{"provingJoId":"1:BASE_PARITY:0e449421da62e439"}
10:20:16 [10:20:16.817]
INFO:
prover-client:proving-broker Proving Broker started
10:20:16 [10:20:16.817]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:656d27bd5beeb85d not found
{"provingJobId":"1:BASE_PARITY:656d27bd5beeb85d"}
10:20:16 [10:20:16.818]
INFO:
prover-client:proving-broker Proving Broker started
10:20:16 [10:20:16.819]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58ed55857582704d epochNumber=1
{"provingJobId":"1:BASE_PARITY:58ed55857582704d"}
10:20:16 [10:20:16.820]
INFO:
prover-client:proving-broker Proving Broker started
10:20:16 [10:20:16.820]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76669825fa9e8af3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:76669825fa9e8af3"}
10:20:16 [10:20:16.921]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:76669825fa9e8af3 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:76669825fa9e8af3"}
10:20:16 [10:20:16.928]
INFO:
prover-client:proving-broker Proving Broker started
10:20:16 [10:20:16.929]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5124dada75968a99 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5124dada75968a99"}
10:20:17 [10:20:17.030]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5124dada75968a99 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5124dada75968a99"}
10:20:17 [10:20:17.037]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f6ba40f79d490c21 epochNumber=10
{"provingJobId":"1:BASE_PARITY:f6ba40f79d490c21"}
10:20:17 [10:20:17.058]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:17 [10:20:17.164]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.166]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54843b1d60f85fa2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:54843b1d60f85fa2"}
10:20:17 [10:20:17.341]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:54843b1d60f85fa2 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:54843b1d60f85fa2"}
10:20:17 [10:20:17.370]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.371]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a9fb31b8783f23b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a9fb31b8783f23b"}
10:20:17 [10:20:17.372]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4a9fb31b8783f23b type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:4a9fb31b8783f23b"}
10:20:17 [10:20:17.373]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.374]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5f58aad372b35c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f5f58aad372b35c7"}
10:20:17 [10:20:17.374]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f5f58aad372b35c7 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f5f58aad372b35c7"}
10:20:17 [10:20:17.375]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f5f58aad372b35c7 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:f5f58aad372b35c7"}
10:20:17 [10:20:17.376]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.376]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3608472f3469fc78 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3608472f3469fc78"}
10:20:17 [10:20:17.377]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3608472f3469fc78 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3608472f3469fc78"}
10:20:17 [10:20:17.377]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.378]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98be517e4c418cb1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:98be517e4c418cb1"}
10:20:17 [10:20:17.378]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:98be517e4c418cb1 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:98be517e4c418cb1"}
10:20:17 [10:20:17.379]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:300d944121b88105 epochNumber=3
{"provingJobId":"1:BASE_PARITY:300d944121b88105"}
10:20:17 [10:20:17.403]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:17 [10:20:17.403]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:98be517e4c418cb1 err=test error
{"provingJoId":"1:BASE_PARITY:98be517e4c418cb1"}
10:20:17 [10:20:17.405]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.406]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1968c1c3f8bae01e settled=false
{"provingJobId":"1:BASE_PARITY:1968c1c3f8bae01e","status":"pending"}
10:20:17 [10:20:17.406]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6f6e3d0ef9ab9fbf settled=false
{"provingJobId":"1:BASE_PARITY:6f6e3d0ef9ab9fbf","status":"pending"}
10:20:17 [10:20:17.409]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.409]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0984518b7f9f57cb settled=true
{"provingJobId":"1:BASE_PARITY:0984518b7f9f57cb","status":"fulfilled"}
10:20:17 [10:20:17.410]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:6673417bd5c084c6 settled=true
{"provingJobId":"2:BASE_PARITY:6673417bd5c084c6","status":"fulfilled"}
10:20:17 [10:20:17.411]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.411]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1ebca58e0ca71bd1 settled=true
{"provingJobId":"1:BASE_PARITY:1ebca58e0ca71bd1","status":"fulfilled"}
10:20:17 [10:20:17.412]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e9309bde2f063b7b settled=false
{"provingJobId":"1:BASE_PARITY:e9309bde2f063b7b","status":"pending"}
10:20:17 [10:20:17.413]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.414]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f06d5299c1133203 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f06d5299c1133203"}
10:20:17 [10:20:17.416]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.416]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff6714cccdb12743 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff6714cccdb12743"}
10:20:17 [10:20:17.417]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:ff6714cccdb12743: Error: db error: Error: db error
10:20:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:ff6714cccdb12743"}
10:20:17 [10:20:17.418]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.418]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:422197aaa2f2c127 epochNumber=1
{"provingJobId":"1:BASE_PARITY:422197aaa2f2c127"}
10:20:17 [10:20:17.419]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:422197aaa2f2c127 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:422197aaa2f2c127"}
10:20:17 [10:20:17.419]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:422197aaa2f2c127 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:422197aaa2f2c127"}
10:20:17 [10:20:17.420]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.421]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01e71c77067ead31 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01e71c77067ead31"}
10:20:17 [10:20:17.421]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:01e71c77067ead31 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:01e71c77067ead31"}
10:20:17 [10:20:17.421]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:01e71c77067ead31 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:01e71c77067ead31"}
10:20:17 [10:20:17.422]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:01e71c77067ead31: Error: db error
10:20:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:01e71c77067ead31"}
10:20:17 [10:20:17.423]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.423]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0aa4075557111b49 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0aa4075557111b49"}
10:20:17 [10:20:17.424]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0aa4075557111b49 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0aa4075557111b49"}
10:20:17 [10:20:17.424]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0aa4075557111b49 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0aa4075557111b49"}
10:20:17 [10:20:17.425]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.426]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d66ca1a4d119c909 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d66ca1a4d119c909"}
10:20:17 [10:20:17.426]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d66ca1a4d119c909 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d66ca1a4d119c909"}
10:20:17 [10:20:17.426]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d66ca1a4d119c909 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d66ca1a4d119c909"}
10:20:17 [10:20:17.427]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:d66ca1a4d119c909 jobErr=test error: Error: db error
10:20:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:d66ca1a4d119c909"}
10:20:17 [10:20:17.428]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.428]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a5108d3dc601adbb not found
{"provingJobId":"1:BASE_PARITY:a5108d3dc601adbb"}
10:20:17 [10:20:17.430]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.430]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b160cca89588060b err=test error
{"provingJoId":"1:BASE_PARITY:b160cca89588060b"}
10:20:17 [10:20:17.432]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.442]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f3bdbba34f5b21a epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f3bdbba34f5b21a"}
10:20:17 [10:20:17.443]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4f3bdbba34f5b21a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4f3bdbba34f5b21a"}
10:20:17 [10:20:17.443]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4f3bdbba34f5b21a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4f3bdbba34f5b21a"}
10:20:17 [10:20:17.454]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:b6514525650e57ff epochNumber=2
{"provingJobId":"2:BASE_PARITY:b6514525650e57ff"}
10:20:17 [10:20:17.454]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:b6514525650e57ff type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:b6514525650e57ff"}
10:20:17 [10:20:17.455]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:b6514525650e57ff type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:b6514525650e57ff"}
10:20:17 [10:20:17.517]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:d3ba5b801e6fa2ea epochNumber=3
{"provingJobId":"3:BASE_PARITY:d3ba5b801e6fa2ea"}
10:20:17 [10:20:17.533]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:17 [10:20:17.579]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:eafe3a0207f66911 epochNumber=4
{"provingJobId":"4:BASE_PARITY:eafe3a0207f66911"}
10:20:17 [10:20:17.583]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:17 [10:20:17.641]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:29a9439b75e56e38 epochNumber=5
{"provingJobId":"5:BASE_PARITY:29a9439b75e56e38"}
10:20:17 [10:20:17.661]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:17 [10:20:17.698]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.699]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2468e2ebe40fbd17 epochNumber=42
{"provingJobId":"1:BASE_PARITY:2468e2ebe40fbd17"}
10:20:17 [10:20:17.700]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testIP9jPb/42 with map size 1073741824
10:20:17 [10:20:17.705]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:17 [10:20:17.733]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:062a9c0ee7cb910c epochNumber=1
{"provingJobId":"1:BASE_PARITY:062a9c0ee7cb910c"}
10:20:17 [10:20:17.737]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.738]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7aa7171a29cc599 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7aa7171a29cc599"}
10:20:17 [10:20:17.739]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testv56EKH/1 with map size 1073741824
10:20:17 [10:20:17.741]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:17 [10:20:17.759]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5fa746d5438b9f1d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5fa746d5438b9f1d"}
10:20:17 [10:20:17.770]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.771]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5fc534dac5e6add0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5fc534dac5e6add0"}
10:20:17 [10:20:17.771]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNc9dP3/1 with map size 1073741824
10:20:17 [10:20:17.774]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:17 [10:20:17.794]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5fc534dac5e6add0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5fc534dac5e6add0"}
10:20:17 [10:20:17.798]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ebeb60a65178d677 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ebeb60a65178d677"}
10:20:17 [10:20:17.800]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiIwh4x/1 with map size 1073741824
10:20:17 [10:20:17.802]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:17 [10:20:17.820]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ebeb60a65178d677 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ebeb60a65178d677"}
10:20:17 [10:20:17.821]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ebeb60a65178d677 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ebeb60a65178d677"}
10:20:17 [10:20:17.827]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ebeb60a65178d677 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ebeb60a65178d677"}
10:20:17 [10:20:17.831]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.832]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ce8f5bb9cfbd74b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ce8f5bb9cfbd74b"}
10:20:17 [10:20:17.833]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmOoxI4/1 with map size 1073741824
10:20:17 [10:20:17.835]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:17 [10:20:17.857]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.861]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.861]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:291ef204de325e91 epochNumber=1
{"provingJobId":"1:BASE_PARITY:291ef204de325e91"}
10:20:17 [10:20:17.863]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYpRFMO/1 with map size 1073741824
10:20:17 [10:20:17.865]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:17 [10:20:17.882]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:291ef204de325e91
{"provingJobId":"1:BASE_PARITY:291ef204de325e91"}
10:20:17 [10:20:17.882]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:291ef204de325e91 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:291ef204de325e91"}
10:20:17 [10:20:17.882]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:291ef204de325e91 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:291ef204de325e91"}
10:20:17 [10:20:17.891]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.891]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aef892cb7701746e epochNumber=1
{"provingJobId":"1:BASE_PARITY:aef892cb7701746e"}
10:20:17 [10:20:17.892]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf9VI6r/1 with map size 1073741824
10:20:17 [10:20:17.893]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:17 [10:20:17.911]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:aef892cb7701746e
{"provingJobId":"1:BASE_PARITY:aef892cb7701746e"}
10:20:17 [10:20:17.912]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:aef892cb7701746e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:aef892cb7701746e"}
10:20:17 [10:20:17.922]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.923]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca8f664f576c3c97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca8f664f576c3c97"}
10:20:17 [10:20:17.924]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOA1u7Y/1 with map size 1073741824
10:20:17 [10:20:17.931]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:17 [10:20:17.949]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ca8f664f576c3c97 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ca8f664f576c3c97"}
10:20:17 [10:20:17.950]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ca8f664f576c3c97 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ca8f664f576c3c97"}
10:20:17 [10:20:17.960]
INFO:
prover-client:proving-broker Proving Broker started
10:20:17 [10:20:17.960]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7487db32c568715 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7487db32c568715"}
10:20:17 [10:20:17.962]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9OYluJ/1 with map size 1073741824
10:20:17 [10:20:17.965]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:17 [10:20:17.984]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f7487db32c568715 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f7487db32c568715"}
10:20:17 [10:20:17.984]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f7487db32c568715 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f7487db32c568715"}
10:20:18 [10:20:18.000]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.001]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04d112249c40db72 epochNumber=0
{"provingJobId":"1:BASE_PARITY:04d112249c40db72"}
10:20:18 [10:20:18.003]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testnWy0iG/0 with map size 1073741824
10:20:18 [10:20:18.009]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.029]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:04d112249c40db72 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:04d112249c40db72"}
10:20:18 [10:20:18.030]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b753bb46989c72f epochNumber=0
{"provingJobId":"1:BASE_PARITY:8b753bb46989c72f"}
10:20:18 [10:20:18.030]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:04d112249c40db72 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:04d112249c40db72"}
10:20:18 [10:20:18.031]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2683f2c90b136f55 epochNumber=0
{"provingJobId":"1:BASE_PARITY:2683f2c90b136f55"}
10:20:18 [10:20:18.047]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.049]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.050]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e192af1e7b1c29d1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e192af1e7b1c29d1"}
10:20:18 [10:20:18.050]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test05nVpl/2 with map size 1073741824
10:20:18 [10:20:18.051]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.069]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c11296eddab474c epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c11296eddab474c"}
10:20:18 [10:20:18.071]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test05nVpl/1 with map size 1073741824
10:20:18 [10:20:18.073]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.098]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.098]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3011c8ab801190ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:3011c8ab801190ff"}
10:20:18 [10:20:18.099]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1dbApw/1 with map size 1073741824
10:20:18 [10:20:18.103]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.126]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a7ebb46f4ba3407 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a7ebb46f4ba3407"}
10:20:18 [10:20:18.128]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyHyBsR/1 with map size 1073741824
10:20:18 [10:20:18.130]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.148]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:275ec535d24c1374 epochNumber=1
{"provingJobId":"1:BASE_PARITY:275ec535d24c1374"}
10:20:18 [10:20:18.154]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ff6118fd438d979 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6ff6118fd438d979"}
10:20:18 [10:20:18.155]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyHyBsR/2 with map size 1073741824
10:20:18 [10:20:18.158]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.178]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ad4814d68f95b24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0ad4814d68f95b24"}
10:20:18 [10:20:18.189]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.190]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c047eb31cb0cce4e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c047eb31cb0cce4e"}
10:20:18 [10:20:18.191]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcOcTHx/1 with map size 1073741824
10:20:18 [10:20:18.194]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.210]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91339b1b5c4f4855 epochNumber=1
{"provingJobId":"1:BASE_PARITY:91339b1b5c4f4855"}
10:20:18 [10:20:18.216]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2a06c7a307327b3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f2a06c7a307327b3"}
10:20:18 [10:20:18.217]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcOcTHx/2 with map size 1073741824
10:20:18 [10:20:18.219]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25ec7826c018f2fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:25ec7826c018f2fa"}
10:20:18 [10:20:18.245]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.246]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43883f3e2dbb713f epochNumber=1
{"provingJobId":"1:BASE_PARITY:43883f3e2dbb713f"}
10:20:18 [10:20:18.246]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt5X2iL/1 with map size 1073741824
10:20:18 [10:20:18.248]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc543c8469e47433 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc543c8469e47433"}
10:20:18 [10:20:18.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c42537519a503823 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c42537519a503823"}
10:20:18 [10:20:18.285]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testt5X2iL/2 with map size 1073741824
10:20:18 [10:20:18.289]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.308]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01f30992e290b325 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01f30992e290b325"}
10:20:18 [10:20:18.318]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.319]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81563e4c857409ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:81563e4c857409ac"}
10:20:18 [10:20:18.320]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKOPyJO/1 with map size 1073741824
10:20:18 [10:20:18.322]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.344]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c863651b8234135 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c863651b8234135"}
10:20:18 [10:20:18.353]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:81563e4c857409ac type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:81563e4c857409ac"}
10:20:18 [10:20:18.368]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.369]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:526c8c672ce60e67 epochNumber=1
{"provingJobId":"1:BASE_PARITY:526c8c672ce60e67"}
10:20:18 [10:20:18.370]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUuuDi2/1 with map size 1073741824
10:20:18 [10:20:18.382]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.406]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d9452a344374c73 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5d9452a344374c73"}
10:20:18 [10:20:18.412]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:526c8c672ce60e67 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:526c8c672ce60e67"}
10:20:18 [10:20:18.420]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.421]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc44880161ed38fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc44880161ed38fd"}
10:20:18 [10:20:18.421]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJWAUn9/1 with map size 1073741824
10:20:18 [10:20:18.423]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.441]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f92ae5f9100d857 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f92ae5f9100d857"}
10:20:18 [10:20:18.447]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fc44880161ed38fd type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:fc44880161ed38fd"}
10:20:18 [10:20:18.451]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:088df2d21b6a604a epochNumber=1
{"provingJobId":"1:BASE_PARITY:088df2d21b6a604a"}
10:20:18 [10:20:18.453]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2gq5mL/1 with map size 1073741824
10:20:18 [10:20:18.455]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.473]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:088df2d21b6a604a
{"provingJobId":"1:BASE_PARITY:088df2d21b6a604a"}
10:20:18 [10:20:18.474]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:088df2d21b6a604a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:088df2d21b6a604a"}
10:20:18 [10:20:18.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3ebde1706dcf446 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3ebde1706dcf446"}
10:20:18 [10:20:18.491]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:088df2d21b6a604a has already been completed
{"provingJobId":"1:BASE_PARITY:088df2d21b6a604a"}
10:20:18 [10:20:18.499]
INFO:
prover-client:proving-broker Proving Broker started
10:20:18 [10:20:18.499]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cbdb68c172481551 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cbdb68c172481551"}
10:20:18 [10:20:18.500]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHMYOro/1 with map size 1073741824
10:20:18 [10:20:18.502]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:18 [10:20:18.519]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:faeada70f0b349cc epochNumber=2
{"provingJobId":"1:BASE_PARITY:faeada70f0b349cc"}
10:20:18 [10:20:18.520]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testHMYOro/2 with map size 1073741824
10:20:18 [10:20:18.522]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:19 [10:20:19.591]
INFO:
prover-client:proving-broker Proving Broker started
10:20:19 [10:20:19.598]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cbdb68c172481551 settled=false
{"provingJobId":"1:BASE_PARITY:cbdb68c172481551","status":"pending"}
10:20:19 [10:20:19.600]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:faeada70f0b349cc settled=false
{"provingJobId":"1:BASE_PARITY:faeada70f0b349cc","status":"pending"}
10:20:19 [10:20:19.601]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cbdb68c172481551 type=BASE_PARITY startedAt=1750328418540 older agent has taken job
{"provingJobId":"1:BASE_PARITY:cbdb68c172481551"}
10:20:19 [10:20:19.601]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cbdb68c172481551 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:cbdb68c172481551"}
10:20:19 [10:20:19.605]
INFO:
prover-client:proving-broker Proving Broker started
10:20:19 [10:20:19.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee67947a90d01c31 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee67947a90d01c31"}
10:20:19 [10:20:19.609]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkjNycY/1 with map size 1073741824
10:20:19 [10:20:19.610]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:19 [10:20:19.629]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce8f25bc4c8cf0c0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ce8f25bc4c8cf0c0"}
10:20:19 [10:20:19.630]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkjNycY/2 with map size 1073741824
10:20:19 [10:20:19.633]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:20 [10:20:20.655]
INFO:
prover-client:proving-broker Proving Broker started
10:20:20 [10:20:20.657]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ee67947a90d01c31 settled=false
{"provingJobId":"1:BASE_PARITY:ee67947a90d01c31","status":"pending"}
10:20:20 [10:20:20.659]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ce8f25bc4c8cf0c0 settled=false
{"provingJobId":"1:BASE_PARITY:ce8f25bc4c8cf0c0","status":"pending"}
10:20:20 [10:20:20.660]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ee67947a90d01c31 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:ee67947a90d01c31"}
10:20:20 [10:20:20.664]
INFO:
prover-client:proving-broker Proving Broker started
10:20:20 [10:20:20.665]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7953842fe608db02 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7953842fe608db02"}
10:20:20 [10:20:20.665]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMKgCjS/1 with map size 1073741824
10:20:20 [10:20:20.667]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:20 [10:20:20.692]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef84038697c753de epochNumber=2
{"provingJobId":"1:BASE_PARITY:ef84038697c753de"}
10:20:20 [10:20:20.693]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMKgCjS/2 with map size 1073741824
10:20:20 [10:20:20.695]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:21 [10:20:21.713]
INFO:
prover-client:proving-broker Proving Broker started
10:20:21 [10:20:21.715]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7953842fe608db02 settled=false
{"provingJobId":"1:BASE_PARITY:7953842fe608db02","status":"pending"}
10:20:21 [10:20:21.717]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ef84038697c753de settled=false
{"provingJobId":"1:BASE_PARITY:ef84038697c753de","status":"pending"}
10:20:21 [10:20:21.718]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7953842fe608db02 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7953842fe608db02"}
10:20:21 [10:20:21.718]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7953842fe608db02 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7953842fe608db02"}
10:20:21 [10:20:21.728]
INFO:
prover-client:proving-broker Proving Broker started
10:20:21 [10:20:21.729]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:07bc43b96c0b7c16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:07bc43b96c0b7c16"}
10:20:21 [10:20:21.731]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAv6Jc7/1 with map size 1073741824
10:20:21 [10:20:21.733]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:21 [10:20:21.755]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1371ca7a163b790f epochNumber=2
{"provingJobId":"1:BASE_PARITY:1371ca7a163b790f"}
10:20:21 [10:20:21.756]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAv6Jc7/2 with map size 1073741824
10:20:21 [10:20:21.758]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:21 [10:20:21.778]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:07bc43b96c0b7c16 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:07bc43b96c0b7c16"}
10:20:21 [10:20:21.785]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1371ca7a163b790f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1371ca7a163b790f"}
10:20:21 [10:20:21.798]
INFO:
prover-client:proving-broker Proving Broker started
10:20:21 [10:20:21.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aac7c0a5348545b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aac7c0a5348545b6"}
10:20:21 [10:20:21.799]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testA2dtMW/1 with map size 1073741824
10:20:21 [10:20:21.801]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:21 [10:20:21.819]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a5fd34f1040fcc60 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a5fd34f1040fcc60"}
10:20:21 [10:20:21.821]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testA2dtMW/2 with map size 1073741824
10:20:21 [10:20:21.823]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:21 [10:20:21.846]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aac7c0a5348545b6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:aac7c0a5348545b6"}
10:20:21 [10:20:21.846]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:aac7c0a5348545b6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:aac7c0a5348545b6"}
10:20:21 [10:20:21.852]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a5fd34f1040fcc60 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a5fd34f1040fcc60"}
10:20:21 [10:20:21.864]
INFO:
prover-client:proving-broker Proving Broker started
10:20:21 [10:20:21.865]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0db8714e56f921a9 err=test error
{"provingJoId":"1:BASE_PARITY:0db8714e56f921a9"}
10:20:21 [10:20:21.867]
INFO:
prover-client:proving-broker Proving Broker started
10:20:21 [10:20:21.868]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:32261d3e2d48a0dd not found
{"provingJobId":"1:BASE_PARITY:32261d3e2d48a0dd"}
10:20:21 [10:20:21.873]
INFO:
prover-client:proving-broker Proving Broker started
10:20:21 [10:20:21.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41ad1b85a5825639 epochNumber=1
{"provingJobId":"1:BASE_PARITY:41ad1b85a5825639"}
10:20:21 [10:20:21.874]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5EdrzV/1 with map size 1073741824
10:20:21 [10:20:21.876]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:21 [10:20:21.900]
INFO:
prover-client:proving-broker Proving Broker started
10:20:21 [10:20:21.901]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c358a96bbcb2b663 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c358a96bbcb2b663"}
10:20:21 [10:20:21.901]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVOBkNR/1 with map size 1073741824
10:20:21 [10:20:21.905]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.026]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c358a96bbcb2b663 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c358a96bbcb2b663"}
10:20:22 [10:20:22.034]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.035]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b3ef7902025a54e epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b3ef7902025a54e"}
10:20:22 [10:20:22.036]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6VTnbe/1 with map size 1073741824
10:20:22 [10:20:22.039]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.162]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2b3ef7902025a54e timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2b3ef7902025a54e"}
10:20:22 [10:20:22.165]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d24b90cefcfded3 epochNumber=10
{"provingJobId":"1:BASE_PARITY:4d24b90cefcfded3"}
10:20:22 [10:20:22.165]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test6VTnbe/10 with map size 1073741824
10:20:22 [10:20:22.169]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.188]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:22 [10:20:22.189]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:20:22 [10:20:22.314]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.315]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:470daff92a9e061a epochNumber=1
{"provingJobId":"1:BASE_PARITY:470daff92a9e061a"}
10:20:22 [10:20:22.316]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYFlGxf/1 with map size 1073741824
10:20:22 [10:20:22.318]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.493]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:470daff92a9e061a timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:470daff92a9e061a"}
10:20:22 [10:20:22.543]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c8d633a3ad013e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c8d633a3ad013e2"}
10:20:22 [10:20:22.544]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuNBOmF/1 with map size 1073741824
10:20:22 [10:20:22.548]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.567]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2c8d633a3ad013e2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:2c8d633a3ad013e2"}
10:20:22 [10:20:22.575]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.576]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6087f2b8b68794a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6087f2b8b68794a7"}
10:20:22 [10:20:22.577]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIxZZWM/1 with map size 1073741824
10:20:22 [10:20:22.579]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.599]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6087f2b8b68794a7 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6087f2b8b68794a7"}
10:20:22 [10:20:22.600]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6087f2b8b68794a7 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:6087f2b8b68794a7"}
10:20:22 [10:20:22.612]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.613]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:693841628edad81e epochNumber=1
{"provingJobId":"1:BASE_PARITY:693841628edad81e"}
10:20:22 [10:20:22.614]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtDBTjg/1 with map size 1073741824
10:20:22 [10:20:22.617]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.638]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:693841628edad81e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:693841628edad81e"}
10:20:22 [10:20:22.648]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.649]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c93027ed12b3653 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c93027ed12b3653"}
10:20:22 [10:20:22.650]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfXIgcU/1 with map size 1073741824
10:20:22 [10:20:22.652]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.680]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5c93027ed12b3653 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:5c93027ed12b3653"}
10:20:22 [10:20:22.682]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50f7db5778924da5 epochNumber=3
{"provingJobId":"1:BASE_PARITY:50f7db5778924da5"}
10:20:22 [10:20:22.683]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testfXIgcU/3 with map size 1073741824
10:20:22 [10:20:22.685]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.725]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:22 [10:20:22.726]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:20:22 [10:20:22.730]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:5c93027ed12b3653 err=test error
{"provingJoId":"1:BASE_PARITY:5c93027ed12b3653"}
10:20:22 [10:20:22.739]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwb9xj0/1 with map size 1073741824
10:20:22 [10:20:22.741]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.762]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwb9xj0/2 with map size 1073741824
10:20:22 [10:20:22.765]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.791]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.792]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8bcd5915d4bc7be1 settled=false
{"provingJobId":"1:BASE_PARITY:8bcd5915d4bc7be1","status":"pending"}
10:20:22 [10:20:22.795]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4223b5f5b8666b8e settled=false
{"provingJobId":"1:BASE_PARITY:4223b5f5b8666b8e","status":"pending"}
10:20:22 [10:20:22.800]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVxkyeU/1 with map size 1073741824
10:20:22 [10:20:22.803]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.821]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVxkyeU/2 with map size 1073741824
10:20:22 [10:20:22.823]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.863]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.865]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c009581e064fb9ca settled=true
{"provingJobId":"1:BASE_PARITY:c009581e064fb9ca","status":"fulfilled"}
10:20:22 [10:20:22.866]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:010b8f853a479620 settled=true
{"provingJobId":"2:BASE_PARITY:010b8f853a479620","status":"fulfilled"}
10:20:22 [10:20:22.872]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwqF8lQ/1 with map size 1073741824
10:20:22 [10:20:22.874]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.906]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwqF8lQ/2 with map size 1073741824
10:20:22 [10:20:22.910]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.932]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.935]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:75d7831ebe4e474a settled=true
{"provingJobId":"1:BASE_PARITY:75d7831ebe4e474a","status":"fulfilled"}
10:20:22 [10:20:22.938]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1ea16decd2279215 settled=false
{"provingJobId":"1:BASE_PARITY:1ea16decd2279215","status":"pending"}
10:20:22 [10:20:22.946]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.947]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21205bcb2a71c7f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:21205bcb2a71c7f8"}
10:20:22 [10:20:22.948]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWbFSy5/1 with map size 1073741824
10:20:22 [10:20:22.951]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:22 [10:20:22.977]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.978]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b412377f47ad6849 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b412377f47ad6849"}
10:20:22 [10:20:22.979]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:b412377f47ad6849: Error: db error: Error: db error
10:20:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:b412377f47ad6849"}
10:20:22 [10:20:22.981]
INFO:
prover-client:proving-broker Proving Broker started
10:20:22 [10:20:22.982]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a51ae6e5d1b5a48e epochNumber=1
{"provingJobId":"1:BASE_PARITY:a51ae6e5d1b5a48e"}
10:20:22 [10:20:22.983]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZ43gy5/1 with map size 1073741824
10:20:22 [10:20:22.986]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:23 [10:20:23.013]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a51ae6e5d1b5a48e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a51ae6e5d1b5a48e"}
10:20:23 [10:20:23.013]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a51ae6e5d1b5a48e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a51ae6e5d1b5a48e"}
10:20:23 [10:20:23.025]
INFO:
prover-client:proving-broker Proving Broker started
10:20:23 [10:20:23.026]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6d0afc8523eb9c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6d0afc8523eb9c4"}
10:20:23 [10:20:23.027]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testK16sq6/1 with map size 1073741824
10:20:23 [10:20:23.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:23 [10:20:23.053]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c6d0afc8523eb9c4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c6d0afc8523eb9c4"}
10:20:23 [10:20:23.054]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c6d0afc8523eb9c4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c6d0afc8523eb9c4"}
10:20:23 [10:20:23.055]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:c6d0afc8523eb9c4: Error: db error
10:20:23 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:c6d0afc8523eb9c4"}
10:20:23 [10:20:23.059]
INFO:
prover-client:proving-broker Proving Broker started
10:20:23 [10:20:23.060]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec99aa8dfaeaff66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec99aa8dfaeaff66"}
10:20:23 [10:20:23.061]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testV0kA6R/1 with map size 1073741824
10:20:23 [10:20:23.063]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:23 [10:20:23.084]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ec99aa8dfaeaff66 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ec99aa8dfaeaff66"}
10:20:23 [10:20:23.085]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ec99aa8dfaeaff66 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ec99aa8dfaeaff66"}
10:20:23 [10:20:23.100]
INFO:
prover-client:proving-broker Proving Broker started
10:20:23 [10:20:23.101]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c3468fe302a8eb47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c3468fe302a8eb47"}
10:20:23 [10:20:23.102]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNvsyeU/1 with map size 1073741824
10:20:23 [10:20:23.105]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:23 [10:20:23.132]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c3468fe302a8eb47 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c3468fe302a8eb47"}
10:20:23 [10:20:23.132]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c3468fe302a8eb47 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c3468fe302a8eb47"}
10:20:23 [10:20:23.133]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:c3468fe302a8eb47 jobErr=test error: Error: db error
10:20:23 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:c3468fe302a8eb47"}
10:20:23 [10:20:23.138]
INFO:
prover-client:proving-broker Proving Broker started
10:20:23 [10:20:23.138]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5dce0e1821099951 not found
{"provingJobId":"1:BASE_PARITY:5dce0e1821099951"}
10:20:23 [10:20:23.143]
INFO:
prover-client:proving-broker Proving Broker started
10:20:23 [10:20:23.144]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8143ddd141bef404 err=test error
{"provingJoId":"1:BASE_PARITY:8143ddd141bef404"}
10:20:23 [10:20:23.148]
INFO:
prover-client:proving-broker Proving Broker started
10:20:23 [10:20:23.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d210bcc12245e04 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d210bcc12245e04"}
10:20:23 [10:20:23.161]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtUXsRq/1 with map size 1073741824
10:20:23 [10:20:23.165]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:23 [10:20:23.190]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0d210bcc12245e04 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0d210bcc12245e04"}
10:20:23 [10:20:23.191]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0d210bcc12245e04 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0d210bcc12245e04"}
10:20:23 [10:20:23.210]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:9c18b56a7bbf7f41 epochNumber=2
{"provingJobId":"2:BASE_PARITY:9c18b56a7bbf7f41"}
10:20:23 [10:20:23.212]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtUXsRq/2 with map size 1073741824
10:20:23 [10:20:23.218]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:23 [10:20:23.242]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:9c18b56a7bbf7f41 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:9c18b56a7bbf7f41"}
10:20:23 [10:20:23.242]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:9c18b56a7bbf7f41 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:9c18b56a7bbf7f41"}
10:20:23 [10:20:23.315]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:31fcb399890507ee epochNumber=3
{"provingJobId":"3:BASE_PARITY:31fcb399890507ee"}
10:20:23 [10:20:23.318]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testtUXsRq/3 with map size 1073741824
10:20:23 [10:20:23.322]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:23 [10:20:23.360]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:23 [10:20:23.360]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:20:23 [10:20:23.405]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:2d172aa201fa62a4 epochNumber=4
{"provingJobId":"4:BASE_PARITY:2d172aa201fa62a4"}
10:20:23 [10:20:23.409]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testtUXsRq/4 with map size 1073741824
10:20:23 [10:20:23.417]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:23 [10:20:23.470]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:23 [10:20:23.470]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
10:20:23 [10:20:23.505]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:b7e3299beaf67f1e epochNumber=5
{"provingJobId":"5:BASE_PARITY:b7e3299beaf67f1e"}
10:20:23 [10:20:23.508]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testtUXsRq/5 with map size 1073741824
10:20:23 [10:20:23.515]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:20:23 [10:20:23.554]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:20:23 [10:20:23.555]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
10:20:23
PASS src/proving_broker/proving_broker.test.ts (
13.189 s)
10:20:23 ProvingBroker
10:20:23 Producer API
10:20:23
✓ refuses stale jobs (58 ms)
10:20:23
✓ enqueues jobs (5 ms)
10:20:23
✓ ignores duplicate jobs (4 ms)
10:20:23
✓ reports correct status when enqueuing repeat jobs (4 ms)
10:20:23
✓ throws an error in case of duplicate job IDs (29 ms)
10:20:23
✓ returns not-found status for non-existing jobs (2 ms)
10:20:23
✓ cancels jobs in queue (5 ms)
10:20:23
✓ cancels jobs in-progress (5 ms)
10:20:23
✓ returns job result if successful (5 ms)
10:20:23
✓ returns job error if failed (6 ms)
10:20:23
✓ correctly returns job status for concurrent writes (5 ms)
10:20:23
✓ refuses stale jobs (41 ms)
10:20:23
✓ enqueues jobs (32 ms)
10:20:23
✓ ignores duplicate jobs (28 ms)
10:20:23
✓ reports correct status when enqueuing repeat jobs (32 ms)
10:20:23
✓ throws an error in case of duplicate job IDs (26 ms)
10:20:23
✓ returns not-found status for non-existing jobs (4 ms)
10:20:23
✓ cancels jobs in queue (30 ms)
10:20:23
✓ cancels jobs in-progress (31 ms)
10:20:23
✓ returns job result if successful (38 ms)
10:20:23
✓ returns job error if failed (40 ms)
10:20:23
✓ correctly returns job status for concurrent writes (47 ms)
10:20:23 Consumer API
10:20:23
✓ returns undefined if no jobs are available (2 ms)
10:20:23
✓ returns jobs in priority order (3 ms)
10:20:23
✓ returns undefined if no jobs are available for the given allowList (2 ms)
10:20:23
✓ returns a job if it is in the allowList (2 ms)
10:20:23
✓ returns the most important job if it is in the allowList (2 ms)
10:20:23
✓ returns any job if filter is empty (3 ms)
10:20:23
✓ returns a new job when reporting job success (2 ms)
10:20:23
✓ returns a new job when reporting permanent error (2 ms)
10:20:23
✓ returns a new job when reporting retry-able error (2 ms)
10:20:23
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
10:20:23
✓ returns a new job if job is already in progress elsewhere (1059 ms)
10:20:23
✓ avoids sending the same job to a new agent after a restart (1004 ms)
10:20:23
✓ avoids sending a completed job to a new agent after a restart (1004 ms)
10:20:23
✓ tracks job result if in progress (2 ms)
10:20:23
✓ tracks job result even if job is in queue (3 ms)
10:20:23
✓ ignores reported job error if unknown job (2 ms)
10:20:23
✓ ignores job result if unknown job (1 ms)
10:20:23
✓ returns undefined if no jobs are available (2 ms)
10:20:23
✓ returns jobs in priority order (48 ms)
10:20:23
✓ returns undefined if no jobs are available for the given allowList (28 ms)
10:20:23
✓ returns a job if it is in the allowList (62 ms)
10:20:23
✓ returns the most important job if it is in the allowList (56 ms)
10:20:23
✓ returns any job if filter is empty (72 ms)
10:20:23
✓ returns a new job when reporting job success (45 ms)
10:20:23
✓ returns a new job when reporting permanent error (55 ms)
10:20:23
✓ returns a new job when reporting retry-able error (29 ms)
10:20:23
✓ returns a new job when reporting progress if current one is cancelled (48 ms)
10:20:23
✓ returns a new job if job is already in progress elsewhere (1106 ms)
10:20:23
✓ avoids sending the same job to a new agent after a restart (1057 ms)
10:20:23
✓ avoids sending a completed job to a new agent after a restart (1064 ms)
10:20:23
✓ tracks job result if in progress (69 ms)
10:20:23
✓ tracks job result even if job is in queue (66 ms)
10:20:23
✓ ignores reported job error if unknown job (3 ms)
10:20:23
✓ ignores job result if unknown job (3 ms)
10:20:23 Timeouts
10:20:23
✓ tracks in progress jobs (1 ms)
10:20:23
✓ re-enqueues jobs that time out (107 ms)
10:20:23
✓ cancel stale jobs that time out (236 ms)
10:20:23
✓ keeps the jobs in progress while it is alive (205 ms)
10:20:23
✓ tracks in progress jobs (27 ms)
10:20:23
✓ re-enqueues jobs that time out (134 ms)
10:20:23
✓ cancel stale jobs that time out (282 ms)
10:20:23
✓ keeps the jobs in progress while it is alive (227 ms)
10:20:23 Retries
10:20:23
✓ retries jobs (3 ms)
10:20:23
✓ retries up to a maximum number of times (2 ms)
10:20:23
✓ passing retry=false does not retry (2 ms)
10:20:23
✓ does not retry if job is stale (27 ms)
10:20:23
✓ retries jobs (31 ms)
10:20:23
✓ retries up to a maximum number of times (38 ms)
10:20:23
✓ passing retry=false does not retry (36 ms)
10:20:23
✓ does not retry if job is stale (86 ms)
10:20:23 Database management
10:20:23
✓ re-enqueues proof requests on start (3 ms)
10:20:23
✓ restores proof results on start (2 ms)
10:20:23
✓ only re-enqueues unfinished jobs (1 ms)
10:20:23
✓ saves job when enqueued (2 ms)
10:20:23
✓ does not retain job if database fails to save (3 ms)
10:20:23
✓ saves job result (2 ms)
10:20:23
✓ saves result even if database fails to save (2 ms)
10:20:23
✓ saves job error (2 ms)
10:20:23
✓ saves job error even if database fails to save (2 ms)
10:20:23
✓ does not save job result if job is unknown (2 ms)
10:20:23
✓ does not save job error if job is unknown (2 ms)
10:20:23
✓ cleans up old jobs periodically (262 ms)
10:20:23
✓ re-enqueues proof requests on start (64 ms)
10:20:23
✓ restores proof results on start (70 ms)
10:20:23
✓ only re-enqueues unfinished jobs (72 ms)
10:20:23
✓ saves job when enqueued (33 ms)
10:20:23
✓ does not retain job if database fails to save (4 ms)
10:20:23
✓ saves job result (42 ms)
10:20:23
✓ saves result even if database fails to save (35 ms)
10:20:23
✓ saves job error (39 ms)
10:20:23
✓ saves job error even if database fails to save (39 ms)
10:20:23
✓ does not save job result if job is unknown (5 ms)
10:20:23
✓ does not save job error if job is unknown (5 ms)
10:20:23
✓ cleans up old jobs periodically (453 ms)
10:20:23
10:20:23
Test Suites: 1 passed, 1 total
10:20:23
Tests: 96 passed, 96 total
10:20:23
Snapshots: 0 total
10:20:23
Time: 13.306 s
10:20:23
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
10:20:23
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?