Parent Log: http://ci.aztec-labs.com/42a2d2f4dc585cb7 Command: 47d4755eef6e3136 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/faf44858d06df712229f99ddabe6e78682dd8a35 Env: REF_NAME=bump-noir CURRENT_VERSION=0.87.6 CI_FULL=0 Date: Wed Jun 18 10:21:36 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=bump-noir_amd64 Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 10:21:39 [10:21:39.324] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.331] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ea7262801fb574a epochNumber=42 {"provingJobId":"1:BASE_PARITY:8ea7262801fb574a"} 10:21:39 [10:21:39.334] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:fba1b25a9941263d epochNumber=1 {"provingJobId":"1:BASE_PARITY:fba1b25a9941263d"} 10:21:39 [10:21:39.360] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.361] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3ba11e4ac5576a03 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3ba11e4ac5576a03"} 10:21:39 [10:21:39.361] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:55b25762f30e9a77 epochNumber=1 {"provingJobId":"1:BASE_PARITY:55b25762f30e9a77"} 10:21:39 [10:21:39.363] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.363] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a9400c701b83d60c epochNumber=1 {"provingJobId":"1:BASE_PARITY:a9400c701b83d60c"} 10:21:39 [10:21:39.364] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a9400c701b83d60c epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:a9400c701b83d60c"} 10:21:39 [10:21:39.365] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.366] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:272f2924b51d3237 epochNumber=1 {"provingJobId":"1:BASE_PARITY:272f2924b51d3237"} 10:21:39 [10:21:39.367] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:272f2924b51d3237 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:272f2924b51d3237"} 10:21:39 [10:21:39.367] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:272f2924b51d3237 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:272f2924b51d3237"} 10:21:39 [10:21:39.368] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:272f2924b51d3237 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:272f2924b51d3237"} 10:21:39 [10:21:39.369] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.369] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f52aa489fcfa9395 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f52aa489fcfa9395"} 10:21:39 [10:21:39.387] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.388] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.388] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:046ea95a991a03c7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:046ea95a991a03c7"} 10:21:39 [10:21:39.389] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:046ea95a991a03c7 {"provingJobId":"1:BASE_PARITY:046ea95a991a03c7"} 10:21:39 [10:21:39.389] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:046ea95a991a03c7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:046ea95a991a03c7"} 10:21:39 [10:21:39.389] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:046ea95a991a03c7 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:046ea95a991a03c7"} 10:21:39 [10:21:39.390] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.391] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4edf9fbf3b767ece epochNumber=1 {"provingJobId":"1:BASE_PARITY:4edf9fbf3b767ece"} 10:21:39 [10:21:39.391] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4edf9fbf3b767ece {"provingJobId":"1:BASE_PARITY:4edf9fbf3b767ece"} 10:21:39 [10:21:39.391] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4edf9fbf3b767ece type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:4edf9fbf3b767ece"} 10:21:39 [10:21:39.392] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.393] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b5198af2b5395aad epochNumber=1 {"provingJobId":"1:BASE_PARITY:b5198af2b5395aad"} 10:21:39 [10:21:39.393] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b5198af2b5395aad type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b5198af2b5395aad"} 10:21:39 [10:21:39.393] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b5198af2b5395aad type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b5198af2b5395aad"} 10:21:39 [10:21:39.394] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.394] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e36484771bf82673 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e36484771bf82673"} 10:21:39 [10:21:39.395] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e36484771bf82673 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e36484771bf82673"} 10:21:39 [10:21:39.395] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e36484771bf82673 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:e36484771bf82673"} 10:21:39 [10:21:39.401] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.401] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b93b3286610198ae epochNumber=0 {"provingJobId":"1:BASE_PARITY:b93b3286610198ae"} 10:21:39 [10:21:39.402] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b93b3286610198ae epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b93b3286610198ae"} 10:21:39 [10:21:39.402] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5238e9f7f82cd094 epochNumber=0 {"provingJobId":"1:BASE_PARITY:5238e9f7f82cd094"} 10:21:39 [10:21:39.402] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b93b3286610198ae epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b93b3286610198ae"} 10:21:39 [10:21:39.402] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6bbdabdf3665db36 epochNumber=0 {"provingJobId":"1:BASE_PARITY:6bbdabdf3665db36"} 10:21:39 [10:21:39.403] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.404] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.404] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:35e9c4dd4d38b9c0 epochNumber=2 {"provingJobId":"1:BASE_PARITY:35e9c4dd4d38b9c0"} 10:21:39 [10:21:39.404] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:51a7206b5ed04889 epochNumber=1 {"provingJobId":"1:BASE_PARITY:51a7206b5ed04889"} 10:21:39 [10:21:39.405] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.406] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4121279580dd90dc epochNumber=1 {"provingJobId":"1:BASE_PARITY:4121279580dd90dc"} 10:21:39 [10:21:39.407] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.407] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f8050966e7864717 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f8050966e7864717"} 10:21:39 [10:21:39.407] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d28ec96b3472b1ec epochNumber=1 {"provingJobId":"1:BASE_PARITY:d28ec96b3472b1ec"} 10:21:39 [10:21:39.408] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1d6a0dde2df7f37e epochNumber=2 {"provingJobId":"1:BASE_PARITY:1d6a0dde2df7f37e"} 10:21:39 [10:21:39.408] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8efbae7e35d40e24 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8efbae7e35d40e24"} 10:21:39 [10:21:39.409] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.409] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9a106e05e3280837 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9a106e05e3280837"} 10:21:39 [10:21:39.409] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:506d6b834c18b231 epochNumber=1 {"provingJobId":"1:BASE_PARITY:506d6b834c18b231"} 10:21:39 [10:21:39.410] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e3383b384e902c17 epochNumber=2 {"provingJobId":"1:BASE_PARITY:e3383b384e902c17"} 10:21:39 [10:21:39.410] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7b39c5454d913cc1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7b39c5454d913cc1"} 10:21:39 [10:21:39.411] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.411] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5602350e382205e5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5602350e382205e5"} 10:21:39 [10:21:39.412] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:080a4a3cb39db680 epochNumber=1 {"provingJobId":"1:BASE_PARITY:080a4a3cb39db680"} 10:21:39 [10:21:39.412] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bc2684f3dc3dd485 epochNumber=2 {"provingJobId":"1:BASE_PARITY:bc2684f3dc3dd485"} 10:21:39 [10:21:39.412] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:801bb792f6005c96 epochNumber=1 {"provingJobId":"1:BASE_PARITY:801bb792f6005c96"} 10:21:39 [10:21:39.413] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.413] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:757ac79713b0c773 epochNumber=1 {"provingJobId":"1:BASE_PARITY:757ac79713b0c773"} 10:21:39 [10:21:39.414] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:57a2a8280309d1da epochNumber=1 {"provingJobId":"1:BASE_PARITY:57a2a8280309d1da"} 10:21:39 [10:21:39.414] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:757ac79713b0c773 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:757ac79713b0c773"} 10:21:39 [10:21:39.415] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.415] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:62092a1585f162fa epochNumber=1 {"provingJobId":"1:BASE_PARITY:62092a1585f162fa"} 10:21:39 [10:21:39.416] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:68621584899a609d epochNumber=1 {"provingJobId":"1:BASE_PARITY:68621584899a609d"} 10:21:39 [10:21:39.416] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:62092a1585f162fa type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:62092a1585f162fa"} 10:21:39 [10:21:39.416] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.417] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:59a488c89d3c94fd epochNumber=1 {"provingJobId":"1:BASE_PARITY:59a488c89d3c94fd"} 10:21:39 [10:21:39.417] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5c2773100786f7f2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5c2773100786f7f2"} 10:21:39 [10:21:39.417] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:59a488c89d3c94fd type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:59a488c89d3c94fd"} 10:21:39 [10:21:39.418] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.418] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9866c9f4e7f7e793 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9866c9f4e7f7e793"} 10:21:39 [10:21:39.418] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9866c9f4e7f7e793 {"provingJobId":"1:BASE_PARITY:9866c9f4e7f7e793"} 10:21:39 [10:21:39.419] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9866c9f4e7f7e793 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:9866c9f4e7f7e793"} 10:21:39 [10:21:39.419] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:acb6e219872944e2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:acb6e219872944e2"} 10:21:39 [10:21:39.419] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9866c9f4e7f7e793 has already been completed {"provingJobId":"1:BASE_PARITY:9866c9f4e7f7e793"} 10:21:39 [10:21:39.420] INFO: prover-client:proving-broker Proving Broker started 10:21:39 [10:21:39.420] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cdc6397b01da64ca epochNumber=1 {"provingJobId":"1:BASE_PARITY:cdc6397b01da64ca"} 10:21:39 [10:21:39.421] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2cd1b463ef8214cc epochNumber=2 {"provingJobId":"1:BASE_PARITY:2cd1b463ef8214cc"} 10:21:40 [10:21:40.471] INFO: prover-client:proving-broker Proving Broker started 10:21:40 [10:21:40.472] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cdc6397b01da64ca settled=false {"provingJobId":"1:BASE_PARITY:cdc6397b01da64ca","status":"pending"} 10:21:40 [10:21:40.473] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2cd1b463ef8214cc settled=false {"provingJobId":"1:BASE_PARITY:2cd1b463ef8214cc","status":"pending"} 10:21:40 [10:21:40.474] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:cdc6397b01da64ca type=BASE_PARITY startedAt=1750242099421 older agent has taken job {"provingJobId":"1:BASE_PARITY:cdc6397b01da64ca"} 10:21:40 [10:21:40.474] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cdc6397b01da64ca type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:cdc6397b01da64ca"} 10:21:40 [10:21:40.475] INFO: prover-client:proving-broker Proving Broker started 10:21:40 [10:21:40.476] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e4a22d68ad1afa0d epochNumber=1 {"provingJobId":"1:BASE_PARITY:e4a22d68ad1afa0d"} 10:21:40 [10:21:40.477] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8b9f3377beceb8b2 epochNumber=2 {"provingJobId":"1:BASE_PARITY:8b9f3377beceb8b2"} 10:21:41 [10:21:41.477] INFO: prover-client:proving-broker Proving Broker started 10:21:41 [10:21:41.478] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e4a22d68ad1afa0d settled=false {"provingJobId":"1:BASE_PARITY:e4a22d68ad1afa0d","status":"pending"} 10:21:41 [10:21:41.478] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8b9f3377beceb8b2 settled=false {"provingJobId":"1:BASE_PARITY:8b9f3377beceb8b2","status":"pending"} 10:21:41 [10:21:41.479] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e4a22d68ad1afa0d type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:e4a22d68ad1afa0d"} 10:21:41 [10:21:41.480] INFO: prover-client:proving-broker Proving Broker started 10:21:41 [10:21:41.481] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:63d5febd629c15bb epochNumber=1 {"provingJobId":"1:BASE_PARITY:63d5febd629c15bb"} 10:21:41 [10:21:41.481] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f922940b748e177e epochNumber=2 {"provingJobId":"1:BASE_PARITY:f922940b748e177e"} 10:21:42 [10:21:42.482] INFO: prover-client:proving-broker Proving Broker started 10:21:42 [10:21:42.482] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:63d5febd629c15bb settled=false {"provingJobId":"1:BASE_PARITY:63d5febd629c15bb","status":"pending"} 10:21:42 [10:21:42.483] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f922940b748e177e settled=false {"provingJobId":"1:BASE_PARITY:f922940b748e177e","status":"pending"} 10:21:42 [10:21:42.484] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:63d5febd629c15bb type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:63d5febd629c15bb"} 10:21:42 [10:21:42.484] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:63d5febd629c15bb type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:63d5febd629c15bb"} 10:21:42 [10:21:42.485] INFO: prover-client:proving-broker Proving Broker started 10:21:42 [10:21:42.486] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e3041c7e730bbc54 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e3041c7e730bbc54"} 10:21:42 [10:21:42.486] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7d85532f9a73ef07 epochNumber=2 {"provingJobId":"1:BASE_PARITY:7d85532f9a73ef07"} 10:21:42 [10:21:42.487] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e3041c7e730bbc54 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e3041c7e730bbc54"} 10:21:42 [10:21:42.488] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7d85532f9a73ef07 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:7d85532f9a73ef07"} 10:21:42 [10:21:42.489] INFO: prover-client:proving-broker Proving Broker started 10:21:42 [10:21:42.489] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab01f77b6e2ff087 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ab01f77b6e2ff087"} 10:21:42 [10:21:42.489] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e8197fbdfeca476c epochNumber=2 {"provingJobId":"1:BASE_PARITY:e8197fbdfeca476c"} 10:21:42 [10:21:42.489] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ab01f77b6e2ff087 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ab01f77b6e2ff087"} 10:21:42 [10:21:42.490] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ab01f77b6e2ff087 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ab01f77b6e2ff087"} 10:21:42 [10:21:42.490] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e8197fbdfeca476c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:e8197fbdfeca476c"} 10:21:42 [10:21:42.491] INFO: prover-client:proving-broker Proving Broker started 10:21:42 [10:21:42.492] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4d3347aed1830070 err=test error {"provingJoId":"1:BASE_PARITY:4d3347aed1830070"} 10:21:42 [10:21:42.493] INFO: prover-client:proving-broker Proving Broker started 10:21:42 [10:21:42.495] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bf88bada2b5a991d not found {"provingJobId":"1:BASE_PARITY:bf88bada2b5a991d"} 10:21:42 [10:21:42.496] INFO: prover-client:proving-broker Proving Broker started 10:21:42 [10:21:42.496] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0f45c01c6ccb53af epochNumber=1 {"provingJobId":"1:BASE_PARITY:0f45c01c6ccb53af"} 10:21:42 [10:21:42.497] INFO: prover-client:proving-broker Proving Broker started 10:21:42 [10:21:42.498] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ff1b86c3516b72fe epochNumber=1 {"provingJobId":"1:BASE_PARITY:ff1b86c3516b72fe"} 10:21:42 [10:21:42.599] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ff1b86c3516b72fe timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:ff1b86c3516b72fe"} 10:21:42 [10:21:42.606] INFO: prover-client:proving-broker Proving Broker started 10:21:42 [10:21:42.607] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d7b6c46dd4fce247 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d7b6c46dd4fce247"} 10:21:42 [10:21:42.707] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d7b6c46dd4fce247 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:d7b6c46dd4fce247"} 10:21:42 [10:21:42.714] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c1a467e464012558 epochNumber=10 {"provingJobId":"1:BASE_PARITY:c1a467e464012558"} 10:21:42 [10:21:42.734] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:42 [10:21:42.840] INFO: prover-client:proving-broker Proving Broker started 10:21:42 [10:21:42.841] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:89433b5badef78b6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:89433b5badef78b6"} 10:21:42 [10:21:42.995] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:89433b5badef78b6 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:89433b5badef78b6"} 10:21:43 [10:21:43.046] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.046] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:be7f262e7c93351b epochNumber=1 {"provingJobId":"1:BASE_PARITY:be7f262e7c93351b"} 10:21:43 [10:21:43.047] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:be7f262e7c93351b type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:be7f262e7c93351b"} 10:21:43 [10:21:43.048] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.048] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b45996cda035dba7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b45996cda035dba7"} 10:21:43 [10:21:43.049] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b45996cda035dba7 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:b45996cda035dba7"} 10:21:43 [10:21:43.049] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b45996cda035dba7 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:b45996cda035dba7"} 10:21:43 [10:21:43.050] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.050] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:22b527f7164fe874 epochNumber=1 {"provingJobId":"1:BASE_PARITY:22b527f7164fe874"} 10:21:43 [10:21:43.050] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:22b527f7164fe874 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:22b527f7164fe874"} 10:21:43 [10:21:43.051] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.051] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:376613dcefe401a9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:376613dcefe401a9"} 10:21:43 [10:21:43.052] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:376613dcefe401a9 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:376613dcefe401a9"} 10:21:43 [10:21:43.052] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:313eb7152bf81bfd epochNumber=3 {"provingJobId":"1:BASE_PARITY:313eb7152bf81bfd"} 10:21:43 [10:21:43.076] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:43 [10:21:43.078] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:376613dcefe401a9 err=test error {"provingJoId":"1:BASE_PARITY:376613dcefe401a9"} 10:21:43 [10:21:43.079] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.079] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3e5ec0c8e260209b settled=false {"provingJobId":"1:BASE_PARITY:3e5ec0c8e260209b","status":"pending"} 10:21:43 [10:21:43.079] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:647a91c526e46ce5 settled=false {"provingJobId":"1:BASE_PARITY:647a91c526e46ce5","status":"pending"} 10:21:43 [10:21:43.081] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.081] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:057398174ac0006a settled=true {"provingJobId":"1:BASE_PARITY:057398174ac0006a","status":"fulfilled"} 10:21:43 [10:21:43.081] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:27f588396f881edf settled=true {"provingJobId":"2:BASE_PARITY:27f588396f881edf","status":"fulfilled"} 10:21:43 [10:21:43.082] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.082] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fb6d0a5aad7a4488 settled=true {"provingJobId":"1:BASE_PARITY:fb6d0a5aad7a4488","status":"fulfilled"} 10:21:43 [10:21:43.082] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6cfacbf99834523f settled=false {"provingJobId":"1:BASE_PARITY:6cfacbf99834523f","status":"pending"} 10:21:43 [10:21:43.083] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.085] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:85bae36b410e8291 epochNumber=1 {"provingJobId":"1:BASE_PARITY:85bae36b410e8291"} 10:21:43 [10:21:43.087] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.088] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dd7b22a2b7b425e4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dd7b22a2b7b425e4"} 10:21:43 [10:21:43.088] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:dd7b22a2b7b425e4: Error: db error: Error: db error 10:21:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:dd7b22a2b7b425e4"} 10:21:43 [10:21:43.090] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.090] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0e429cd56aafb6e8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0e429cd56aafb6e8"} 10:21:43 [10:21:43.090] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0e429cd56aafb6e8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0e429cd56aafb6e8"} 10:21:43 [10:21:43.090] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0e429cd56aafb6e8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0e429cd56aafb6e8"} 10:21:43 [10:21:43.091] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.092] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e6d335f7c42fbdee epochNumber=1 {"provingJobId":"1:BASE_PARITY:e6d335f7c42fbdee"} 10:21:43 [10:21:43.092] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e6d335f7c42fbdee type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e6d335f7c42fbdee"} 10:21:43 [10:21:43.092] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e6d335f7c42fbdee type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e6d335f7c42fbdee"} 10:21:43 [10:21:43.092] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:e6d335f7c42fbdee: Error: db error 10:21:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:e6d335f7c42fbdee"} 10:21:43 [10:21:43.093] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.093] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:37e58d49f447bb76 epochNumber=1 {"provingJobId":"1:BASE_PARITY:37e58d49f447bb76"} 10:21:43 [10:21:43.093] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:37e58d49f447bb76 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:37e58d49f447bb76"} 10:21:43 [10:21:43.094] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:37e58d49f447bb76 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:37e58d49f447bb76"} 10:21:43 [10:21:43.094] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.095] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:176601f2c078f734 epochNumber=1 {"provingJobId":"1:BASE_PARITY:176601f2c078f734"} 10:21:43 [10:21:43.095] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:176601f2c078f734 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:176601f2c078f734"} 10:21:43 [10:21:43.095] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:176601f2c078f734 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:176601f2c078f734"} 10:21:43 [10:21:43.095] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:176601f2c078f734 jobErr=test error: Error: db error 10:21:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:176601f2c078f734"} 10:21:43 [10:21:43.096] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.096] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:15199c03c25c1722 not found {"provingJobId":"1:BASE_PARITY:15199c03c25c1722"} 10:21:43 [10:21:43.098] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.098] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e92d6b48ad1c1773 err=test error {"provingJoId":"1:BASE_PARITY:e92d6b48ad1c1773"} 10:21:43 [10:21:43.099] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.110] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:304232229aedda8a epochNumber=1 {"provingJobId":"1:BASE_PARITY:304232229aedda8a"} 10:21:43 [10:21:43.111] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:304232229aedda8a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:304232229aedda8a"} 10:21:43 [10:21:43.111] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:304232229aedda8a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:304232229aedda8a"} 10:21:43 [10:21:43.121] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:15ed7b792d4d88b7 epochNumber=2 {"provingJobId":"2:BASE_PARITY:15ed7b792d4d88b7"} 10:21:43 [10:21:43.122] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:15ed7b792d4d88b7 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:15ed7b792d4d88b7"} 10:21:43 [10:21:43.122] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:15ed7b792d4d88b7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:15ed7b792d4d88b7"} 10:21:43 [10:21:43.182] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:c1c3ec6515279416 epochNumber=3 {"provingJobId":"3:BASE_PARITY:c1c3ec6515279416"} 10:21:43 [10:21:43.202] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:43 [10:21:43.244] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:dd7b98708d055f6f epochNumber=4 {"provingJobId":"4:BASE_PARITY:dd7b98708d055f6f"} 10:21:43 [10:21:43.254] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:43 [10:21:43.306] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:5a0888610450f017 epochNumber=5 {"provingJobId":"5:BASE_PARITY:5a0888610450f017"} 10:21:43 [10:21:43.330] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:43 [10:21:43.360] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.361] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:27d1f488a036077a epochNumber=42 {"provingJobId":"1:BASE_PARITY:27d1f488a036077a"} 10:21:43 [10:21:43.362] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testRQg14v/42 with map size 1073741824 10:21:43 [10:21:43.365] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.392] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:c4439abac3aa07e4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c4439abac3aa07e4"} 10:21:43 [10:21:43.396] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.397] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ee8fb4992dd6cef3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ee8fb4992dd6cef3"} 10:21:43 [10:21:43.397] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testD4FDIH/1 with map size 1073741824 10:21:43 [10:21:43.398] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.416] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5d9c3db83c057d70 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5d9c3db83c057d70"} 10:21:43 [10:21:43.425] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.426] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1847114258e61435 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1847114258e61435"} 10:21:43 [10:21:43.426] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwyjKlf/1 with map size 1073741824 10:21:43 [10:21:43.428] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.446] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1847114258e61435 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:1847114258e61435"} 10:21:43 [10:21:43.449] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.450] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d5527e5b55f4a11e epochNumber=1 {"provingJobId":"1:BASE_PARITY:d5527e5b55f4a11e"} 10:21:43 [10:21:43.451] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeGNc82/1 with map size 1073741824 10:21:43 [10:21:43.452] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.468] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d5527e5b55f4a11e epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:d5527e5b55f4a11e"} 10:21:43 [10:21:43.468] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d5527e5b55f4a11e type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d5527e5b55f4a11e"} 10:21:43 [10:21:43.474] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d5527e5b55f4a11e epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:d5527e5b55f4a11e"} 10:21:43 [10:21:43.477] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.478] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:faf78319a054613e epochNumber=1 {"provingJobId":"1:BASE_PARITY:faf78319a054613e"} 10:21:43 [10:21:43.478] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQ0SaCp/1 with map size 1073741824 10:21:43 [10:21:43.480] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.501] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.504] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.504] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:00b7c3ffd6c21ca4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:00b7c3ffd6c21ca4"} 10:21:43 [10:21:43.505] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testToNXqb/1 with map size 1073741824 10:21:43 [10:21:43.506] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.525] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:00b7c3ffd6c21ca4 {"provingJobId":"1:BASE_PARITY:00b7c3ffd6c21ca4"} 10:21:43 [10:21:43.526] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:00b7c3ffd6c21ca4 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:00b7c3ffd6c21ca4"} 10:21:43 [10:21:43.526] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:00b7c3ffd6c21ca4 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:00b7c3ffd6c21ca4"} 10:21:43 [10:21:43.535] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.535] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae594b60744bad32 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ae594b60744bad32"} 10:21:43 [10:21:43.536] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrSrtNn/1 with map size 1073741824 10:21:43 [10:21:43.537] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.556] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ae594b60744bad32 {"provingJobId":"1:BASE_PARITY:ae594b60744bad32"} 10:21:43 [10:21:43.557] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ae594b60744bad32 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:ae594b60744bad32"} 10:21:43 [10:21:43.566] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.567] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5ea7334d6dd7eee5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5ea7334d6dd7eee5"} 10:21:43 [10:21:43.567] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRgpEn1/1 with map size 1073741824 10:21:43 [10:21:43.569] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.587] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5ea7334d6dd7eee5 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5ea7334d6dd7eee5"} 10:21:43 [10:21:43.587] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5ea7334d6dd7eee5 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5ea7334d6dd7eee5"} 10:21:43 [10:21:43.595] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.595] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e4d04e5d6dc6e202 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e4d04e5d6dc6e202"} 10:21:43 [10:21:43.596] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testscNDq2/1 with map size 1073741824 10:21:43 [10:21:43.598] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.614] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e4d04e5d6dc6e202 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e4d04e5d6dc6e202"} 10:21:43 [10:21:43.615] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e4d04e5d6dc6e202 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:e4d04e5d6dc6e202"} 10:21:43 [10:21:43.622] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.623] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e0c44f2b3f29f456 epochNumber=0 {"provingJobId":"1:BASE_PARITY:e0c44f2b3f29f456"} 10:21:43 [10:21:43.623] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testmOBYOg/0 with map size 1073741824 10:21:43 [10:21:43.625] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.640] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e0c44f2b3f29f456 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:e0c44f2b3f29f456"} 10:21:43 [10:21:43.640] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f8a65c7154f89c27 epochNumber=0 {"provingJobId":"1:BASE_PARITY:f8a65c7154f89c27"} 10:21:43 [10:21:43.641] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e0c44f2b3f29f456 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:e0c44f2b3f29f456"} 10:21:43 [10:21:43.641] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:661d8b9d4410582e epochNumber=0 {"provingJobId":"1:BASE_PARITY:661d8b9d4410582e"} 10:21:43 [10:21:43.653] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.654] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.655] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aae3b316507fcfff epochNumber=2 {"provingJobId":"1:BASE_PARITY:aae3b316507fcfff"} 10:21:43 [10:21:43.655] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFZvzwI/2 with map size 1073741824 10:21:43 [10:21:43.656] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.671] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0d2fd4ec35f85cd0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0d2fd4ec35f85cd0"} 10:21:43 [10:21:43.672] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFZvzwI/1 with map size 1073741824 10:21:43 [10:21:43.673] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.692] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.693] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7ca3f2f6fef636c9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7ca3f2f6fef636c9"} 10:21:43 [10:21:43.693] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4rr5kQ/1 with map size 1073741824 10:21:43 [10:21:43.695] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.714] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.714] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:02478958540a6ce4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:02478958540a6ce4"} 10:21:43 [10:21:43.715] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testB3ca81/1 with map size 1073741824 10:21:43 [10:21:43.717] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.733] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae93c32acf0c8508 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ae93c32acf0c8508"} 10:21:43 [10:21:43.739] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab8169f83f7f8169 epochNumber=2 {"provingJobId":"1:BASE_PARITY:ab8169f83f7f8169"} 10:21:43 [10:21:43.740] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testB3ca81/2 with map size 1073741824 10:21:43 [10:21:43.742] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.759] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:668dcf95460fcb12 epochNumber=1 {"provingJobId":"1:BASE_PARITY:668dcf95460fcb12"} 10:21:43 [10:21:43.767] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.768] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6870dd8e480a23a3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6870dd8e480a23a3"} 10:21:43 [10:21:43.768] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrabcd0/1 with map size 1073741824 10:21:43 [10:21:43.770] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.786] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:42fecb9e7268cbf7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:42fecb9e7268cbf7"} 10:21:43 [10:21:43.792] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c0dd43a70baf74e7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:c0dd43a70baf74e7"} 10:21:43 [10:21:43.792] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrabcd0/2 with map size 1073741824 10:21:43 [10:21:43.794] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.810] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6760a7201209672d epochNumber=1 {"provingJobId":"1:BASE_PARITY:6760a7201209672d"} 10:21:43 [10:21:43.819] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.820] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d693494d8af9fcd7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d693494d8af9fcd7"} 10:21:43 [10:21:43.821] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFGkWVi/1 with map size 1073741824 10:21:43 [10:21:43.823] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.839] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:77127779981ab3a6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:77127779981ab3a6"} 10:21:43 [10:21:43.845] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:194eaa3e7c174646 epochNumber=2 {"provingJobId":"1:BASE_PARITY:194eaa3e7c174646"} 10:21:43 [10:21:43.849] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFGkWVi/2 with map size 1073741824 10:21:43 [10:21:43.851] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.867] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8d28faf5eddc79c5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8d28faf5eddc79c5"} 10:21:43 [10:21:43.875] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.876] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c1de69249813d19f epochNumber=1 {"provingJobId":"1:BASE_PARITY:c1de69249813d19f"} 10:21:43 [10:21:43.877] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5PgsqD/1 with map size 1073741824 10:21:43 [10:21:43.878] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.894] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b8242b94cfdde4b2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b8242b94cfdde4b2"} 10:21:43 [10:21:43.899] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c1de69249813d19f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:c1de69249813d19f"} 10:21:43 [10:21:43.907] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.908] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7133e8db3d08ffe0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7133e8db3d08ffe0"} 10:21:43 [10:21:43.908] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwFqK6r/1 with map size 1073741824 10:21:43 [10:21:43.910] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.924] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bc6ea14075f5a4c9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bc6ea14075f5a4c9"} 10:21:43 [10:21:43.930] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7133e8db3d08ffe0 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:7133e8db3d08ffe0"} 10:21:43 [10:21:43.938] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.939] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3981af955eb0a1f1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3981af955eb0a1f1"} 10:21:43 [10:21:43.939] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYu77rx/1 with map size 1073741824 10:21:43 [10:21:43.941] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.957] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c5f75292555905ea epochNumber=1 {"provingJobId":"1:BASE_PARITY:c5f75292555905ea"} 10:21:43 [10:21:43.966] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3981af955eb0a1f1 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:3981af955eb0a1f1"} 10:21:43 [10:21:43.969] INFO: prover-client:proving-broker Proving Broker started 10:21:43 [10:21:43.970] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2a1e43b69475bc61 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2a1e43b69475bc61"} 10:21:43 [10:21:43.971] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxXJNKR/1 with map size 1073741824 10:21:43 [10:21:43.972] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:43 [10:21:43.993] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2a1e43b69475bc61 {"provingJobId":"1:BASE_PARITY:2a1e43b69475bc61"} 10:21:43 [10:21:43.993] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2a1e43b69475bc61 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:2a1e43b69475bc61"} 10:21:43 [10:21:43.998] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab8a095acef7e55d epochNumber=1 {"provingJobId":"1:BASE_PARITY:ab8a095acef7e55d"} 10:21:44 [10:21:44.002] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2a1e43b69475bc61 has already been completed {"provingJobId":"1:BASE_PARITY:2a1e43b69475bc61"} 10:21:44 [10:21:44.005] INFO: prover-client:proving-broker Proving Broker started 10:21:44 [10:21:44.006] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e0faeb3bb061a3d1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e0faeb3bb061a3d1"} 10:21:44 [10:21:44.006] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7oCYjV/1 with map size 1073741824 10:21:44 [10:21:44.007] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:44 [10:21:44.022] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:189131206c4a807d epochNumber=2 {"provingJobId":"1:BASE_PARITY:189131206c4a807d"} 10:21:44 [10:21:44.023] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7oCYjV/2 with map size 1073741824 10:21:44 [10:21:44.025] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:45 [10:21:45.091] INFO: prover-client:proving-broker Proving Broker started 10:21:45 [10:21:45.096] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e0faeb3bb061a3d1 settled=false {"provingJobId":"1:BASE_PARITY:e0faeb3bb061a3d1","status":"pending"} 10:21:45 [10:21:45.098] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:189131206c4a807d settled=false {"provingJobId":"1:BASE_PARITY:189131206c4a807d","status":"pending"} 10:21:45 [10:21:45.099] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:e0faeb3bb061a3d1 type=BASE_PARITY startedAt=1750242104039 older agent has taken job {"provingJobId":"1:BASE_PARITY:e0faeb3bb061a3d1"} 10:21:45 [10:21:45.099] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e0faeb3bb061a3d1 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:e0faeb3bb061a3d1"} 10:21:45 [10:21:45.103] INFO: prover-client:proving-broker Proving Broker started 10:21:45 [10:21:45.103] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:57a58df0dd3ca4cc epochNumber=1 {"provingJobId":"1:BASE_PARITY:57a58df0dd3ca4cc"} 10:21:45 [10:21:45.105] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhEBPEs/1 with map size 1073741824 10:21:45 [10:21:45.109] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:45 [10:21:45.129] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4666048273ace49a epochNumber=2 {"provingJobId":"1:BASE_PARITY:4666048273ace49a"} 10:21:45 [10:21:45.130] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testhEBPEs/2 with map size 1073741824 10:21:45 [10:21:45.133] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:46 [10:21:46.150] INFO: prover-client:proving-broker Proving Broker started 10:21:46 [10:21:46.152] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:57a58df0dd3ca4cc settled=false {"provingJobId":"1:BASE_PARITY:57a58df0dd3ca4cc","status":"pending"} 10:21:46 [10:21:46.153] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4666048273ace49a settled=false {"provingJobId":"1:BASE_PARITY:4666048273ace49a","status":"pending"} 10:21:46 [10:21:46.154] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:57a58df0dd3ca4cc type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:57a58df0dd3ca4cc"} 10:21:46 [10:21:46.158] INFO: prover-client:proving-broker Proving Broker started 10:21:46 [10:21:46.159] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bab730af2d415aae epochNumber=1 {"provingJobId":"1:BASE_PARITY:bab730af2d415aae"} 10:21:46 [10:21:46.159] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testT2BlVW/1 with map size 1073741824 10:21:46 [10:21:46.161] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:46 [10:21:46.179] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2fef15071022629b epochNumber=2 {"provingJobId":"1:BASE_PARITY:2fef15071022629b"} 10:21:46 [10:21:46.180] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testT2BlVW/2 with map size 1073741824 10:21:46 [10:21:46.182] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.199] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.201] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bab730af2d415aae settled=false {"provingJobId":"1:BASE_PARITY:bab730af2d415aae","status":"pending"} 10:21:47 [10:21:47.203] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2fef15071022629b settled=false {"provingJobId":"1:BASE_PARITY:2fef15071022629b","status":"pending"} 10:21:47 [10:21:47.205] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bab730af2d415aae type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:bab730af2d415aae"} 10:21:47 [10:21:47.205] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bab730af2d415aae type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:bab730af2d415aae"} 10:21:47 [10:21:47.217] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.218] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8e2abfe043c04f4e epochNumber=1 {"provingJobId":"1:BASE_PARITY:8e2abfe043c04f4e"} 10:21:47 [10:21:47.218] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhvQGJM/1 with map size 1073741824 10:21:47 [10:21:47.220] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.236] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03a96b2dd23c7fff epochNumber=2 {"provingJobId":"1:BASE_PARITY:03a96b2dd23c7fff"} 10:21:47 [10:21:47.237] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testhvQGJM/2 with map size 1073741824 10:21:47 [10:21:47.240] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.257] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8e2abfe043c04f4e type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8e2abfe043c04f4e"} 10:21:47 [10:21:47.263] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:03a96b2dd23c7fff type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:03a96b2dd23c7fff"} 10:21:47 [10:21:47.273] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.273] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:31882731684e70fc epochNumber=1 {"provingJobId":"1:BASE_PARITY:31882731684e70fc"} 10:21:47 [10:21:47.274] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt6hisc/1 with map size 1073741824 10:21:47 [10:21:47.276] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.292] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3822f2a671b4e0cf epochNumber=2 {"provingJobId":"1:BASE_PARITY:3822f2a671b4e0cf"} 10:21:47 [10:21:47.293] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testt6hisc/2 with map size 1073741824 10:21:47 [10:21:47.295] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.312] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:31882731684e70fc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:31882731684e70fc"} 10:21:47 [10:21:47.313] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:31882731684e70fc type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:31882731684e70fc"} 10:21:47 [10:21:47.318] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3822f2a671b4e0cf type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3822f2a671b4e0cf"} 10:21:47 [10:21:47.326] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.327] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:679d0c02f6b31733 err=test error {"provingJoId":"1:BASE_PARITY:679d0c02f6b31733"} 10:21:47 [10:21:47.329] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.330] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2e48ea62d9233b1f not found {"provingJobId":"1:BASE_PARITY:2e48ea62d9233b1f"} 10:21:47 [10:21:47.332] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.333] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cce21a688a653065 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cce21a688a653065"} 10:21:47 [10:21:47.333] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFmSRbI/1 with map size 1073741824 10:21:47 [10:21:47.334] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.355] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.356] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:67bb1a959cc948f4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:67bb1a959cc948f4"} 10:21:47 [10:21:47.357] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRcopOV/1 with map size 1073741824 10:21:47 [10:21:47.359] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.481] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:67bb1a959cc948f4 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:67bb1a959cc948f4"} 10:21:47 [10:21:47.490] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.491] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8e9faf3def85300e epochNumber=1 {"provingJobId":"1:BASE_PARITY:8e9faf3def85300e"} 10:21:47 [10:21:47.491] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test15zxLg/1 with map size 1073741824 10:21:47 [10:21:47.493] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.616] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8e9faf3def85300e timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:8e9faf3def85300e"} 10:21:47 [10:21:47.619] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d382f12378edbce1 epochNumber=10 {"provingJobId":"1:BASE_PARITY:d382f12378edbce1"} 10:21:47 [10:21:47.620] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test15zxLg/10 with map size 1073741824 10:21:47 [10:21:47.622] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.642] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:47 [10:21:47.643] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 10:21:47 [10:21:47.767] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.768] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c3bb0f2985f7e235 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c3bb0f2985f7e235"} 10:21:47 [10:21:47.769] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbZy5gl/1 with map size 1073741824 10:21:47 [10:21:47.772] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:47 [10:21:47.944] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c3bb0f2985f7e235 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:c3bb0f2985f7e235"} 10:21:47 [10:21:47.997] INFO: prover-client:proving-broker Proving Broker started 10:21:47 [10:21:47.998] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:88b9de999e35cb00 epochNumber=1 {"provingJobId":"1:BASE_PARITY:88b9de999e35cb00"} 10:21:47 [10:21:47.998] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeRFbzY/1 with map size 1073741824 10:21:48 [10:21:48.002] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.026] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:88b9de999e35cb00 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:88b9de999e35cb00"} 10:21:48 [10:21:48.030] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.031] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bb8c84fab2cf1f84 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bb8c84fab2cf1f84"} 10:21:48 [10:21:48.031] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSgOuQA/1 with map size 1073741824 10:21:48 [10:21:48.034] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.057] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bb8c84fab2cf1f84 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:bb8c84fab2cf1f84"} 10:21:48 [10:21:48.058] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bb8c84fab2cf1f84 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:bb8c84fab2cf1f84"} 10:21:48 [10:21:48.067] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.067] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:43b9d4e33186451f epochNumber=1 {"provingJobId":"1:BASE_PARITY:43b9d4e33186451f"} 10:21:48 [10:21:48.068] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgcvTat/1 with map size 1073741824 10:21:48 [10:21:48.071] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.093] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:43b9d4e33186451f type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:43b9d4e33186451f"} 10:21:48 [10:21:48.105] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.106] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:15c268ccc7df6ed7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:15c268ccc7df6ed7"} 10:21:48 [10:21:48.108] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJ1siBG/1 with map size 1073741824 10:21:48 [10:21:48.111] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.131] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:15c268ccc7df6ed7 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:15c268ccc7df6ed7"} 10:21:48 [10:21:48.133] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4cf8188751b58b40 epochNumber=3 {"provingJobId":"1:BASE_PARITY:4cf8188751b58b40"} 10:21:48 [10:21:48.134] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testJ1siBG/3 with map size 1073741824 10:21:48 [10:21:48.140] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.183] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:48 [10:21:48.184] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 10:21:48 [10:21:48.188] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:15c268ccc7df6ed7 err=test error {"provingJoId":"1:BASE_PARITY:15c268ccc7df6ed7"} 10:21:48 [10:21:48.193] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGU087A/1 with map size 1073741824 10:21:48 [10:21:48.196] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.219] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGU087A/2 with map size 1073741824 10:21:48 [10:21:48.223] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.247] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.248] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:31308214ac28bdc2 settled=false {"provingJobId":"1:BASE_PARITY:31308214ac28bdc2","status":"pending"} 10:21:48 [10:21:48.250] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1117e96adb2d5752 settled=false {"provingJobId":"1:BASE_PARITY:1117e96adb2d5752","status":"pending"} 10:21:48 [10:21:48.254] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiIqsgp/1 with map size 1073741824 10:21:48 [10:21:48.257] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.283] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testiIqsgp/2 with map size 1073741824 10:21:48 [10:21:48.286] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.317] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.319] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d65bda26fb1b9132 settled=true {"provingJobId":"1:BASE_PARITY:d65bda26fb1b9132","status":"fulfilled"} 10:21:48 [10:21:48.321] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:38fa4cb95ab91ec6 settled=true {"provingJobId":"2:BASE_PARITY:38fa4cb95ab91ec6","status":"fulfilled"} 10:21:48 [10:21:48.327] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testV16bnG/1 with map size 1073741824 10:21:48 [10:21:48.328] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.370] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testV16bnG/2 with map size 1073741824 10:21:48 [10:21:48.375] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.399] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.401] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dcc2f830e816eeda settled=true {"provingJobId":"1:BASE_PARITY:dcc2f830e816eeda","status":"fulfilled"} 10:21:48 [10:21:48.404] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9b82027935838cca settled=false {"provingJobId":"1:BASE_PARITY:9b82027935838cca","status":"pending"} 10:21:48 [10:21:48.410] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.411] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d371961883b5eb64 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d371961883b5eb64"} 10:21:48 [10:21:48.411] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testd5Ns2k/1 with map size 1073741824 10:21:48 [10:21:48.413] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.435] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.435] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d98b258c0f2299dd epochNumber=1 {"provingJobId":"1:BASE_PARITY:d98b258c0f2299dd"} 10:21:48 [10:21:48.436] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:d98b258c0f2299dd: Error: db error: Error: db error 10:21:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:d98b258c0f2299dd"} 10:21:48 [10:21:48.439] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.440] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a470de5b22d58230 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a470de5b22d58230"} 10:21:48 [10:21:48.441] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testW4SVjW/1 with map size 1073741824 10:21:48 [10:21:48.445] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.467] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a470de5b22d58230 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a470de5b22d58230"} 10:21:48 [10:21:48.468] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a470de5b22d58230 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:a470de5b22d58230"} 10:21:48 [10:21:48.480] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.481] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:da0fce75dc788f1b epochNumber=1 {"provingJobId":"1:BASE_PARITY:da0fce75dc788f1b"} 10:21:48 [10:21:48.482] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testL2u1z5/1 with map size 1073741824 10:21:48 [10:21:48.483] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.507] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:da0fce75dc788f1b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:da0fce75dc788f1b"} 10:21:48 [10:21:48.508] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:da0fce75dc788f1b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:da0fce75dc788f1b"} 10:21:48 [10:21:48.509] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:da0fce75dc788f1b: Error: db error 10:21:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:da0fce75dc788f1b"} 10:21:48 [10:21:48.515] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.516] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae5f25e50a918f97 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ae5f25e50a918f97"} 10:21:48 [10:21:48.517] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9fsIF2/1 with map size 1073741824 10:21:48 [10:21:48.519] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.543] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ae5f25e50a918f97 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ae5f25e50a918f97"} 10:21:48 [10:21:48.543] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ae5f25e50a918f97 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ae5f25e50a918f97"} 10:21:48 [10:21:48.552] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.553] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ccd3decc26106fee epochNumber=1 {"provingJobId":"1:BASE_PARITY:ccd3decc26106fee"} 10:21:48 [10:21:48.556] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testofEZf6/1 with map size 1073741824 10:21:48 [10:21:48.559] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.580] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ccd3decc26106fee type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ccd3decc26106fee"} 10:21:48 [10:21:48.580] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ccd3decc26106fee type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ccd3decc26106fee"} 10:21:48 [10:21:48.581] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:ccd3decc26106fee jobErr=test error: Error: db error 10:21:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:ccd3decc26106fee"} 10:21:48 [10:21:48.586] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.587] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f1bd498f87cf0328 not found {"provingJobId":"1:BASE_PARITY:f1bd498f87cf0328"} 10:21:48 [10:21:48.590] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.590] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7502d19771c16c19 err=test error {"provingJoId":"1:BASE_PARITY:7502d19771c16c19"} 10:21:48 [10:21:48.593] INFO: prover-client:proving-broker Proving Broker started 10:21:48 [10:21:48.603] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:67a1f2eccc140513 epochNumber=1 {"provingJobId":"1:BASE_PARITY:67a1f2eccc140513"} 10:21:48 [10:21:48.604] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiBmNYz/1 with map size 1073741824 10:21:48 [10:21:48.606] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.631] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:67a1f2eccc140513 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:67a1f2eccc140513"} 10:21:48 [10:21:48.631] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:67a1f2eccc140513 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:67a1f2eccc140513"} 10:21:48 [10:21:48.649] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:443d7dbcba72c8ed epochNumber=2 {"provingJobId":"2:BASE_PARITY:443d7dbcba72c8ed"} 10:21:48 [10:21:48.650] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testiBmNYz/2 with map size 1073741824 10:21:48 [10:21:48.652] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.675] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:443d7dbcba72c8ed type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:443d7dbcba72c8ed"} 10:21:48 [10:21:48.676] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:443d7dbcba72c8ed type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:443d7dbcba72c8ed"} 10:21:48 [10:21:48.743] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:7fb2df91ed9514bc epochNumber=3 {"provingJobId":"3:BASE_PARITY:7fb2df91ed9514bc"} 10:21:48 [10:21:48.745] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testiBmNYz/3 with map size 1073741824 10:21:48 [10:21:48.749] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.795] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:48 [10:21:48.795] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 10:21:48 [10:21:48.836] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:4fab74a345955712 epochNumber=4 {"provingJobId":"4:BASE_PARITY:4fab74a345955712"} 10:21:48 [10:21:48.838] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testiBmNYz/4 with map size 1073741824 10:21:48 [10:21:48.841] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.874] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:48 [10:21:48.875] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 10:21:48 [10:21:48.921] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:0360c72b7c160648 epochNumber=5 {"provingJobId":"5:BASE_PARITY:0360c72b7c160648"} 10:21:48 [10:21:48.922] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testiBmNYz/5 with map size 1073741824 10:21:48 [10:21:48.925] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 10:21:48 [10:21:48.953] INFO: prover-client:proving-broker Cleaned up jobs=1 10:21:48 [10:21:48.954] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 10:21:49 PASS src/proving_broker/proving_broker.test.ts (12.034 s) 10:21:49 ProvingBroker 10:21:49 Producer API 10:21:49 ✓ refuses stale jobs (39 ms) 10:21:49 ✓ enqueues jobs (3 ms) 10:21:49 ✓ ignores duplicate jobs (3 ms) 10:21:49 ✓ reports correct status when enqueuing repeat jobs (3 ms) 10:21:49 ✓ throws an error in case of duplicate job IDs (18 ms) 10:21:49 ✓ returns not-found status for non-existing jobs (1 ms) 10:21:49 ✓ cancels jobs in queue (2 ms) 10:21:49 ✓ cancels jobs in-progress (1 ms) 10:21:49 ✓ returns job result if successful (1 ms) 10:21:49 ✓ returns job error if failed (1 ms) 10:21:49 ✓ correctly returns job status for concurrent writes (8 ms) 10:21:49 ✓ refuses stale jobs (38 ms) 10:21:49 ✓ enqueues jobs (29 ms) 10:21:49 ✓ ignores duplicate jobs (24 ms) 10:21:49 ✓ reports correct status when enqueuing repeat jobs (28 ms) 10:21:49 ✓ throws an error in case of duplicate job IDs (23 ms) 10:21:49 ✓ returns not-found status for non-existing jobs (2 ms) 10:21:49 ✓ cancels jobs in queue (31 ms) 10:21:49 ✓ cancels jobs in-progress (31 ms) 10:21:49 ✓ returns job result if successful (29 ms) 10:21:49 ✓ returns job error if failed (27 ms) 10:21:49 ✓ correctly returns job status for concurrent writes (31 ms) 10:21:49 Consumer API 10:21:49 ✓ returns undefined if no jobs are available (1 ms) 10:21:49 ✓ returns jobs in priority order (1 ms) 10:21:49 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 10:21:49 ✓ returns a job if it is in the allowList (2 ms) 10:21:49 ✓ returns the most important job if it is in the allowList (2 ms) 10:21:49 ✓ returns any job if filter is empty (1 ms) 10:21:49 ✓ returns a new job when reporting job success (2 ms) 10:21:49 ✓ returns a new job when reporting permanent error (1 ms) 10:21:49 ✓ returns a new job when reporting retry-able error (1 ms) 10:21:49 ✓ returns a new job when reporting progress if current one is cancelled (2 ms) 10:21:49 ✓ returns a new job if job is already in progress elsewhere (1055 ms) 10:21:49 ✓ avoids sending the same job to a new agent after a restart (1004 ms) 10:21:49 ✓ avoids sending a completed job to a new agent after a restart (1005 ms) 10:21:49 ✓ tracks job result if in progress (3 ms) 10:21:49 ✓ tracks job result even if job is in queue (3 ms) 10:21:49 ✓ ignores reported job error if unknown job (1 ms) 10:21:49 ✓ ignores job result if unknown job (2 ms) 10:21:49 ✓ returns undefined if no jobs are available (2 ms) 10:21:49 ✓ returns jobs in priority order (36 ms) 10:21:49 ✓ returns undefined if no jobs are available for the given allowList (21 ms) 10:21:49 ✓ returns a job if it is in the allowList (54 ms) 10:21:49 ✓ returns the most important job if it is in the allowList (51 ms) 10:21:49 ✓ returns any job if filter is empty (56 ms) 10:21:49 ✓ returns a new job when reporting job success (32 ms) 10:21:49 ✓ returns a new job when reporting permanent error (30 ms) 10:21:49 ✓ returns a new job when reporting retry-able error (31 ms) 10:21:49 ✓ returns a new job when reporting progress if current one is cancelled (36 ms) 10:21:49 ✓ returns a new job if job is already in progress elsewhere (1097 ms) 10:21:49 ✓ avoids sending the same job to a new agent after a restart (1055 ms) 10:21:49 ✓ avoids sending a completed job to a new agent after a restart (1057 ms) 10:21:49 ✓ tracks job result if in progress (56 ms) 10:21:49 ✓ tracks job result even if job is in queue (53 ms) 10:21:49 ✓ ignores reported job error if unknown job (3 ms) 10:21:49 ✓ ignores job result if unknown job (3 ms) 10:21:49 Timeouts 10:21:49 ✓ tracks in progress jobs (2 ms) 10:21:49 ✓ re-enqueues jobs that time out (108 ms) 10:21:49 ✓ cancel stale jobs that time out (233 ms) 10:21:49 ✓ keeps the jobs in progress while it is alive (206 ms) 10:21:49 ✓ tracks in progress jobs (23 ms) 10:21:49 ✓ re-enqueues jobs that time out (134 ms) 10:21:49 ✓ cancel stale jobs that time out (277 ms) 10:21:49 ✓ keeps the jobs in progress while it is alive (229 ms) 10:21:49 Retries 10:21:49 ✓ retries jobs (2 ms) 10:21:49 ✓ retries up to a maximum number of times (1 ms) 10:21:49 ✓ passing retry=false does not retry (1 ms) 10:21:49 ✓ does not retry if job is stale (27 ms) 10:21:49 ✓ retries jobs (33 ms) 10:21:49 ✓ retries up to a maximum number of times (37 ms) 10:21:49 ✓ passing retry=false does not retry (36 ms) 10:21:49 ✓ does not retry if job is stale (88 ms) 10:21:49 Database management 10:21:49 ✓ re-enqueues proof requests on start (2 ms) 10:21:49 ✓ restores proof results on start (2 ms) 10:21:49 ✓ only re-enqueues unfinished jobs (1 ms) 10:21:49 ✓ saves job when enqueued (3 ms) 10:21:49 ✓ does not retain job if database fails to save (3 ms) 10:21:49 ✓ saves job result (2 ms) 10:21:49 ✓ saves result even if database fails to save (2 ms) 10:21:49 ✓ saves job error (1 ms) 10:21:49 ✓ saves job error even if database fails to save (2 ms) 10:21:49 ✓ does not save job result if job is unknown (1 ms) 10:21:49 ✓ does not save job error if job is unknown (1 ms) 10:21:49 ✓ cleans up old jobs periodically (258 ms) 10:21:49 ✓ re-enqueues proof requests on start (62 ms) 10:21:49 ✓ restores proof results on start (69 ms) 10:21:49 ✓ only re-enqueues unfinished jobs (84 ms) 10:21:49 ✓ saves job when enqueued (25 ms) 10:21:49 ✓ does not retain job if database fails to save (4 ms) 10:21:49 ✓ saves job result (40 ms) 10:21:49 ✓ saves result even if database fails to save (33 ms) 10:21:49 ✓ saves job error (38 ms) 10:21:49 ✓ saves job error even if database fails to save (33 ms) 10:21:49 ✓ does not save job result if job is unknown (5 ms) 10:21:49 ✓ does not save job error if job is unknown (3 ms) 10:21:49 ✓ cleans up old jobs periodically (407 ms) 10:21:49 10:21:49 Test Suites: 1 passed, 1 total 10:21:49 Tests: 96 passed, 96 total 10:21:49 Snapshots: 0 total 10:21:49 Time: 12.106 s 10:21:49 Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i. 10:21:49 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?