Parent Log:
http://ci.aztec-labs.com/461faebffafe60ea
Command: 6e611b2acba6dc44 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/9a37be7dc4208f0eeb44ec50194c00ca3f5c4cc6
Env: REF_NAME=gh-readonly-queue/next/pr-15072-1e338a3fb2e2077f1feaee8b86c42644ff8a5352 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 16 16:41:50 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15072_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
16:41:55 [16:41:55.030]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3881e2c209fe2ea5 epochNumber=42
{"provingJobId":"1:BASE_PARITY:3881e2c209fe2ea5"}
16:41:55 [16:41:55.039]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:39eb57e10a43124d epochNumber=1
{"provingJobId":"1:BASE_PARITY:39eb57e10a43124d"}
16:41:55 [16:41:55.069]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.070]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0aea4e92a66173c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0aea4e92a66173c9"}
16:41:55 [16:41:55.071]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8aecace4387afc7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f8aecace4387afc7"}
16:41:55 [16:41:55.072]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:69382e94301138cc epochNumber=1
{"provingJobId":"1:BASE_PARITY:69382e94301138cc"}
16:41:55 [16:41:55.074]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:69382e94301138cc epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:69382e94301138cc"}
16:41:55 [16:41:55.076]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:249f6b138d991c1b epochNumber=1
{"provingJobId":"1:BASE_PARITY:249f6b138d991c1b"}
16:41:55 [16:41:55.078]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:249f6b138d991c1b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:249f6b138d991c1b"}
16:41:55 [16:41:55.079]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:249f6b138d991c1b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:249f6b138d991c1b"}
16:41:55 [16:41:55.079]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:249f6b138d991c1b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:249f6b138d991c1b"}
16:41:55 [16:41:55.081]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.081]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b17022333875094 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b17022333875094"}
16:41:55 [16:41:55.109]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.110]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.111]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4540ad3f68e86298 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4540ad3f68e86298"}
16:41:55 [16:41:55.112]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4540ad3f68e86298
{"provingJobId":"1:BASE_PARITY:4540ad3f68e86298"}
16:41:55 [16:41:55.112]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4540ad3f68e86298 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4540ad3f68e86298"}
16:41:55 [16:41:55.112]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4540ad3f68e86298 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4540ad3f68e86298"}
16:41:55 [16:41:55.114]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.114]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a201e5bbfde80523 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a201e5bbfde80523"}
16:41:55 [16:41:55.115]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a201e5bbfde80523
{"provingJobId":"1:BASE_PARITY:a201e5bbfde80523"}
16:41:55 [16:41:55.115]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a201e5bbfde80523 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a201e5bbfde80523"}
16:41:55 [16:41:55.117]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.117]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fdc9a3ae437715c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fdc9a3ae437715c9"}
16:41:55 [16:41:55.118]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fdc9a3ae437715c9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fdc9a3ae437715c9"}
16:41:55 [16:41:55.118]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fdc9a3ae437715c9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fdc9a3ae437715c9"}
16:41:55 [16:41:55.119]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.120]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:880bb316734887b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:880bb316734887b1"}
16:41:55 [16:41:55.120]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:880bb316734887b1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:880bb316734887b1"}
16:41:55 [16:41:55.120]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:880bb316734887b1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:880bb316734887b1"}
16:41:55 [16:41:55.121]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9fccc45a59656533 epochNumber=0
{"provingJobId":"1:BASE_PARITY:9fccc45a59656533"}
16:41:55 [16:41:55.122]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9fccc45a59656533 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9fccc45a59656533"}
16:41:55 [16:41:55.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a8ec73544da4dd6a epochNumber=0
{"provingJobId":"1:BASE_PARITY:a8ec73544da4dd6a"}
16:41:55 [16:41:55.122]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9fccc45a59656533 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9fccc45a59656533"}
16:41:55 [16:41:55.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b7d2a6d6ea1253a epochNumber=0
{"provingJobId":"1:BASE_PARITY:1b7d2a6d6ea1253a"}
16:41:55 [16:41:55.123]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.124]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.125]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:465ba398fef702c4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:465ba398fef702c4"}
16:41:55 [16:41:55.125]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:636ecfd1bfe3ddef epochNumber=1
{"provingJobId":"1:BASE_PARITY:636ecfd1bfe3ddef"}
16:41:55 [16:41:55.126]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.126]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a4099b9eba0d812 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a4099b9eba0d812"}
16:41:55 [16:41:55.127]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c7f9bbc19b347ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:7c7f9bbc19b347ba"}
16:41:55 [16:41:55.128]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:764789f0b7180aa9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:764789f0b7180aa9"}
16:41:55 [16:41:55.128]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:437b71472a1d0e59 epochNumber=2
{"provingJobId":"1:BASE_PARITY:437b71472a1d0e59"}
16:41:55 [16:41:55.128]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01f2ef7bb996f1a3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01f2ef7bb996f1a3"}
16:41:55 [16:41:55.129]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.129]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ebc3e4a675a88b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ebc3e4a675a88b1"}
16:41:55 [16:41:55.130]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa352907be0fa856 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa352907be0fa856"}
16:41:55 [16:41:55.130]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f21eeeb6de08a747 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f21eeeb6de08a747"}
16:41:55 [16:41:55.130]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:807a99b900ebe0ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:807a99b900ebe0ea"}
16:41:55 [16:41:55.131]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.132]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c723549536143087 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c723549536143087"}
16:41:55 [16:41:55.132]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:518a5888f96aa2ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:518a5888f96aa2ec"}
16:41:55 [16:41:55.132]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4af9554a52c46da epochNumber=2
{"provingJobId":"1:BASE_PARITY:d4af9554a52c46da"}
16:41:55 [16:41:55.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:247cbc9c6c8c004a epochNumber=1
{"provingJobId":"1:BASE_PARITY:247cbc9c6c8c004a"}
16:41:55 [16:41:55.134]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.134]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6be68d3224de960e epochNumber=1
{"provingJobId":"1:BASE_PARITY:6be68d3224de960e"}
16:41:55 [16:41:55.135]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c1099d17a56e53f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c1099d17a56e53f"}
16:41:55 [16:41:55.135]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6be68d3224de960e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6be68d3224de960e"}
16:41:55 [16:41:55.137]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.137]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32b08a8ed5e689b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:32b08a8ed5e689b6"}
16:41:55 [16:41:55.138]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c128cada22d44ddc epochNumber=1
{"provingJobId":"1:BASE_PARITY:c128cada22d44ddc"}
16:41:55 [16:41:55.138]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:32b08a8ed5e689b6 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:32b08a8ed5e689b6"}
16:41:55 [16:41:55.139]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.140]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1802f8c6a25ddbd7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1802f8c6a25ddbd7"}
16:41:55 [16:41:55.140]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28d0d156fb29510d epochNumber=1
{"provingJobId":"1:BASE_PARITY:28d0d156fb29510d"}
16:41:55 [16:41:55.141]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1802f8c6a25ddbd7 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:1802f8c6a25ddbd7"}
16:41:55 [16:41:55.142]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.142]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f131e7c83695c56 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f131e7c83695c56"}
16:41:55 [16:41:55.142]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6f131e7c83695c56
{"provingJobId":"1:BASE_PARITY:6f131e7c83695c56"}
16:41:55 [16:41:55.143]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6f131e7c83695c56 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:6f131e7c83695c56"}
16:41:55 [16:41:55.143]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5084a16b864c96b7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5084a16b864c96b7"}
16:41:55 [16:41:55.143]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6f131e7c83695c56 has already been completed
{"provingJobId":"1:BASE_PARITY:6f131e7c83695c56"}
16:41:55 [16:41:55.150]
INFO:
prover-client:proving-broker Proving Broker started
16:41:55 [16:41:55.152]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b0309f30497386e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b0309f30497386e"}
16:41:55 [16:41:55.152]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d351b856ba9c2f6f epochNumber=2
{"provingJobId":"1:BASE_PARITY:d351b856ba9c2f6f"}
16:41:56 [16:41:56.205]
INFO:
prover-client:proving-broker Proving Broker started
16:41:56 [16:41:56.205]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4b0309f30497386e settled=false
{"provingJobId":"1:BASE_PARITY:4b0309f30497386e","status":"pending"}
16:41:56 [16:41:56.206]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d351b856ba9c2f6f settled=false
{"provingJobId":"1:BASE_PARITY:d351b856ba9c2f6f","status":"pending"}
16:41:56 [16:41:56.206]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4b0309f30497386e type=BASE_PARITY startedAt=1750092115152 older agent has taken job
{"provingJobId":"1:BASE_PARITY:4b0309f30497386e"}
16:41:56 [16:41:56.207]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4b0309f30497386e type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:4b0309f30497386e"}
16:41:56 [16:41:56.207]
INFO:
prover-client:proving-broker Proving Broker started
16:41:56 [16:41:56.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f01a82a08aba9ef epochNumber=1
{"provingJobId":"1:BASE_PARITY:7f01a82a08aba9ef"}
16:41:56 [16:41:56.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52540ecb160c84f7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:52540ecb160c84f7"}
16:41:57 [16:41:57.212]
INFO:
prover-client:proving-broker Proving Broker started
16:41:57 [16:41:57.212]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7f01a82a08aba9ef settled=false
{"provingJobId":"1:BASE_PARITY:7f01a82a08aba9ef","status":"pending"}
16:41:57 [16:41:57.213]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:52540ecb160c84f7 settled=false
{"provingJobId":"1:BASE_PARITY:52540ecb160c84f7","status":"pending"}
16:41:57 [16:41:57.213]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7f01a82a08aba9ef type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:7f01a82a08aba9ef"}
16:41:57 [16:41:57.215]
INFO:
prover-client:proving-broker Proving Broker started
16:41:57 [16:41:57.215]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4bf2c66fa95eedbb epochNumber=1
{"provingJobId":"1:BASE_PARITY:4bf2c66fa95eedbb"}
16:41:57 [16:41:57.216]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be3cab5663f1e6a3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:be3cab5663f1e6a3"}
16:41:58 [16:41:58.217]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.217]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4bf2c66fa95eedbb settled=false
{"provingJobId":"1:BASE_PARITY:4bf2c66fa95eedbb","status":"pending"}
16:41:58 [16:41:58.217]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:be3cab5663f1e6a3 settled=false
{"provingJobId":"1:BASE_PARITY:be3cab5663f1e6a3","status":"pending"}
16:41:58 [16:41:58.218]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4bf2c66fa95eedbb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4bf2c66fa95eedbb"}
16:41:58 [16:41:58.218]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4bf2c66fa95eedbb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4bf2c66fa95eedbb"}
16:41:58 [16:41:58.219]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.220]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f01ee8dc95121eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f01ee8dc95121eb"}
16:41:58 [16:41:58.220]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a976973335e98d9b epochNumber=2
{"provingJobId":"1:BASE_PARITY:a976973335e98d9b"}
16:41:58 [16:41:58.220]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0f01ee8dc95121eb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0f01ee8dc95121eb"}
16:41:58 [16:41:58.221]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a976973335e98d9b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a976973335e98d9b"}
16:41:58 [16:41:58.222]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10914211fe81d060 epochNumber=1
{"provingJobId":"1:BASE_PARITY:10914211fe81d060"}
16:41:58 [16:41:58.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbcf2a7084e4ed6f epochNumber=2
{"provingJobId":"1:BASE_PARITY:dbcf2a7084e4ed6f"}
16:41:58 [16:41:58.223]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:10914211fe81d060 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:10914211fe81d060"}
16:41:58 [16:41:58.223]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:10914211fe81d060 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:10914211fe81d060"}
16:41:58 [16:41:58.224]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dbcf2a7084e4ed6f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:dbcf2a7084e4ed6f"}
16:41:58 [16:41:58.225]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.225]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8b62360aa168b912 err=test error
{"provingJoId":"1:BASE_PARITY:8b62360aa168b912"}
16:41:58 [16:41:58.226]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.227]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3d9c0b5d29c862e7 not found
{"provingJobId":"1:BASE_PARITY:3d9c0b5d29c862e7"}
16:41:58 [16:41:58.228]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.229]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc590d144344bc7e epochNumber=1
{"provingJobId":"1:BASE_PARITY:bc590d144344bc7e"}
16:41:58 [16:41:58.230]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.231]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:daf3a70c9c7d3fb5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:daf3a70c9c7d3fb5"}
16:41:58 [16:41:58.356]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:daf3a70c9c7d3fb5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:daf3a70c9c7d3fb5"}
16:41:58 [16:41:58.360]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.361]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce513d5eae914926 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ce513d5eae914926"}
16:41:58 [16:41:58.463]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ce513d5eae914926 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:ce513d5eae914926"}
16:41:58 [16:41:58.469]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:716ab0413a801d18 epochNumber=10
{"provingJobId":"1:BASE_PARITY:716ab0413a801d18"}
16:41:58 [16:41:58.489]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:41:58 [16:41:58.596]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.597]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b99cfeabed0d4a18 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b99cfeabed0d4a18"}
16:41:58 [16:41:58.776]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b99cfeabed0d4a18 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b99cfeabed0d4a18"}
16:41:58 [16:41:58.806]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c73a438c09863f27 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c73a438c09863f27"}
16:41:58 [16:41:58.807]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c73a438c09863f27 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c73a438c09863f27"}
16:41:58 [16:41:58.808]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8927c401c1f69bec epochNumber=1
{"provingJobId":"1:BASE_PARITY:8927c401c1f69bec"}
16:41:58 [16:41:58.809]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8927c401c1f69bec type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:8927c401c1f69bec"}
16:41:58 [16:41:58.810]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8927c401c1f69bec type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:8927c401c1f69bec"}
16:41:58 [16:41:58.811]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.811]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7311adf1deca6956 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7311adf1deca6956"}
16:41:58 [16:41:58.811]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7311adf1deca6956 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7311adf1deca6956"}
16:41:58 [16:41:58.812]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c135c15462821ef5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c135c15462821ef5"}
16:41:58 [16:41:58.813]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c135c15462821ef5 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c135c15462821ef5"}
16:41:58 [16:41:58.814]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b8e5cca48682403 epochNumber=3
{"provingJobId":"1:BASE_PARITY:4b8e5cca48682403"}
16:41:58 [16:41:58.837]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:41:58 [16:41:58.840]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c135c15462821ef5 err=test error
{"provingJoId":"1:BASE_PARITY:c135c15462821ef5"}
16:41:58 [16:41:58.842]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.842]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e489251960220862 settled=false
{"provingJobId":"1:BASE_PARITY:e489251960220862","status":"pending"}
16:41:58 [16:41:58.842]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4832fb3bc0fb0781 settled=false
{"provingJobId":"1:BASE_PARITY:4832fb3bc0fb0781","status":"pending"}
16:41:58 [16:41:58.844]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.844]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:780425eda407d979 settled=true
{"provingJobId":"1:BASE_PARITY:780425eda407d979","status":"fulfilled"}
16:41:58 [16:41:58.844]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:9246a1b5dc939e93 settled=true
{"provingJobId":"2:BASE_PARITY:9246a1b5dc939e93","status":"fulfilled"}
16:41:58 [16:41:58.845]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.846]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:284d2c990c78512a settled=true
{"provingJobId":"1:BASE_PARITY:284d2c990c78512a","status":"fulfilled"}
16:41:58 [16:41:58.846]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c23a775f780eae95 settled=false
{"provingJobId":"1:BASE_PARITY:c23a775f780eae95","status":"pending"}
16:41:58 [16:41:58.847]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.848]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5bf2ab9fabd7a16c epochNumber=1
{"provingJobId":"1:BASE_PARITY:5bf2ab9fabd7a16c"}
16:41:58 [16:41:58.849]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.850]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25a073ab0a30d2a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:25a073ab0a30d2a4"}
16:41:58 [16:41:58.850]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:25a073ab0a30d2a4: Error: db error: Error: db error
16:41:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:25a073ab0a30d2a4"}
16:41:58 [16:41:58.851]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.852]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ddb23f0a460bf7e epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ddb23f0a460bf7e"}
16:41:58 [16:41:58.852]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9ddb23f0a460bf7e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9ddb23f0a460bf7e"}
16:41:58 [16:41:58.852]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9ddb23f0a460bf7e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9ddb23f0a460bf7e"}
16:41:58 [16:41:58.853]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.854]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b96570e37d8ced9e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b96570e37d8ced9e"}
16:41:58 [16:41:58.854]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b96570e37d8ced9e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b96570e37d8ced9e"}
16:41:58 [16:41:58.854]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b96570e37d8ced9e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b96570e37d8ced9e"}
16:41:58 [16:41:58.855]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:b96570e37d8ced9e: Error: db error
16:41:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:b96570e37d8ced9e"}
16:41:58 [16:41:58.856]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.856]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64453c021f172e1c epochNumber=1
{"provingJobId":"1:BASE_PARITY:64453c021f172e1c"}
16:41:58 [16:41:58.856]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:64453c021f172e1c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:64453c021f172e1c"}
16:41:58 [16:41:58.857]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:64453c021f172e1c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:64453c021f172e1c"}
16:41:58 [16:41:58.858]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.858]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fd800c03006446b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fd800c03006446b"}
16:41:58 [16:41:58.858]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4fd800c03006446b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4fd800c03006446b"}
16:41:58 [16:41:58.858]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4fd800c03006446b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4fd800c03006446b"}
16:41:58 [16:41:58.859]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:4fd800c03006446b jobErr=test error: Error: db error
16:41:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:4fd800c03006446b"}
16:41:58 [16:41:58.859]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.860]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d3a08fb60b70da87 not found
{"provingJobId":"1:BASE_PARITY:d3a08fb60b70da87"}
16:41:58 [16:41:58.861]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.861]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7c31ce873fd1d34d err=test error
{"provingJoId":"1:BASE_PARITY:7c31ce873fd1d34d"}
16:41:58 [16:41:58.862]
INFO:
prover-client:proving-broker Proving Broker started
16:41:58 [16:41:58.871]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b2de9cb27352387 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b2de9cb27352387"}
16:41:58 [16:41:58.872]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2b2de9cb27352387 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2b2de9cb27352387"}
16:41:58 [16:41:58.872]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2b2de9cb27352387 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2b2de9cb27352387"}
16:41:58 [16:41:58.882]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:5441eafcdea06443 epochNumber=2
{"provingJobId":"2:BASE_PARITY:5441eafcdea06443"}
16:41:58 [16:41:58.883]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:5441eafcdea06443 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:5441eafcdea06443"}
16:41:58 [16:41:58.884]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:5441eafcdea06443 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:5441eafcdea06443"}
16:41:58 [16:41:58.946]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:944f44c30bc0880c epochNumber=3
{"provingJobId":"3:BASE_PARITY:944f44c30bc0880c"}
16:41:58 [16:41:58.964]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:41:59 [16:41:59.008]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:2524b3316d08ca47 epochNumber=4
{"provingJobId":"4:BASE_PARITY:2524b3316d08ca47"}
16:41:59 [16:41:59.015]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:41:59 [16:41:59.070]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:bc0fa2c56221dcff epochNumber=5
{"provingJobId":"5:BASE_PARITY:bc0fa2c56221dcff"}
16:41:59 [16:41:59.090]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:41:59 [16:41:59.125]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.126]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2cb6541b6596645f epochNumber=42
{"provingJobId":"1:BASE_PARITY:2cb6541b6596645f"}
16:41:59 [16:41:59.128]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testTNyKDE/42 with map size 1073741824
16:41:59 [16:41:59.133]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.163]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:63b13e05a6b4b7a2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:63b13e05a6b4b7a2"}
16:41:59 [16:41:59.169]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.170]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a203ee8329d32c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a203ee8329d32c1"}
16:41:59 [16:41:59.171]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZLykxg/1 with map size 1073741824
16:41:59 [16:41:59.172]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.191]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15f4b0abdf68a76f epochNumber=1
{"provingJobId":"1:BASE_PARITY:15f4b0abdf68a76f"}
16:41:59 [16:41:59.204]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.205]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:adbaefaae8117621 epochNumber=1
{"provingJobId":"1:BASE_PARITY:adbaefaae8117621"}
16:41:59 [16:41:59.205]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testC79CAm/1 with map size 1073741824
16:41:59 [16:41:59.207]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.226]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:adbaefaae8117621 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:adbaefaae8117621"}
16:41:59 [16:41:59.229]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.230]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9855a312c68ce396 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9855a312c68ce396"}
16:41:59 [16:41:59.231]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZlim7F/1 with map size 1073741824
16:41:59 [16:41:59.233]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.256]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9855a312c68ce396 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9855a312c68ce396"}
16:41:59 [16:41:59.256]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9855a312c68ce396 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9855a312c68ce396"}
16:41:59 [16:41:59.263]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9855a312c68ce396 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9855a312c68ce396"}
16:41:59 [16:41:59.266]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.267]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4a4f825f36a9196 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4a4f825f36a9196"}
16:41:59 [16:41:59.268]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWNOWAO/1 with map size 1073741824
16:41:59 [16:41:59.270]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.293]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.296]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.296]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19910efea4bf7a9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:19910efea4bf7a9f"}
16:41:59 [16:41:59.297]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testV4W1FP/1 with map size 1073741824
16:41:59 [16:41:59.301]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.322]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:19910efea4bf7a9f
{"provingJobId":"1:BASE_PARITY:19910efea4bf7a9f"}
16:41:59 [16:41:59.322]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:19910efea4bf7a9f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:19910efea4bf7a9f"}
16:41:59 [16:41:59.323]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:19910efea4bf7a9f type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:19910efea4bf7a9f"}
16:41:59 [16:41:59.332]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.332]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:545645270fe911b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:545645270fe911b9"}
16:41:59 [16:41:59.333]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrqgTF9/1 with map size 1073741824
16:41:59 [16:41:59.334]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.352]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:545645270fe911b9
{"provingJobId":"1:BASE_PARITY:545645270fe911b9"}
16:41:59 [16:41:59.352]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:545645270fe911b9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:545645270fe911b9"}
16:41:59 [16:41:59.361]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99de68e096fefe05 epochNumber=1
{"provingJobId":"1:BASE_PARITY:99de68e096fefe05"}
16:41:59 [16:41:59.362]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYDNawZ/1 with map size 1073741824
16:41:59 [16:41:59.364]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.381]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:99de68e096fefe05 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:99de68e096fefe05"}
16:41:59 [16:41:59.381]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:99de68e096fefe05 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:99de68e096fefe05"}
16:41:59 [16:41:59.392]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.393]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9fe1705e0fb286f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9fe1705e0fb286f0"}
16:41:59 [16:41:59.394]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYZIo4L/1 with map size 1073741824
16:41:59 [16:41:59.398]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.418]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9fe1705e0fb286f0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9fe1705e0fb286f0"}
16:41:59 [16:41:59.419]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9fe1705e0fb286f0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9fe1705e0fb286f0"}
16:41:59 [16:41:59.428]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.429]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c14e0657ac77aa2 epochNumber=0
{"provingJobId":"1:BASE_PARITY:7c14e0657ac77aa2"}
16:41:59 [16:41:59.429]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test5zVLv7/0 with map size 1073741824
16:41:59 [16:41:59.431]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.458]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7c14e0657ac77aa2 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7c14e0657ac77aa2"}
16:41:59 [16:41:59.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b764c40e4180903d epochNumber=0
{"provingJobId":"1:BASE_PARITY:b764c40e4180903d"}
16:41:59 [16:41:59.460]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7c14e0657ac77aa2 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7c14e0657ac77aa2"}
16:41:59 [16:41:59.460]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24f8219bb982396a epochNumber=0
{"provingJobId":"1:BASE_PARITY:24f8219bb982396a"}
16:41:59 [16:41:59.468]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.470]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.471]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:837db9a4227b4112 epochNumber=2
{"provingJobId":"1:BASE_PARITY:837db9a4227b4112"}
16:41:59 [16:41:59.472]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testPmMkW3/2 with map size 1073741824
16:41:59 [16:41:59.474]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.483]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fbda4ec3f816e2f1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fbda4ec3f816e2f1"}
16:41:59 [16:41:59.484]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPmMkW3/1 with map size 1073741824
16:41:59 [16:41:59.486]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.499]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:abec2f3ecbbffcab epochNumber=1
{"provingJobId":"1:BASE_PARITY:abec2f3ecbbffcab"}
16:41:59 [16:41:59.500]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfDHjMO/1 with map size 1073741824
16:41:59 [16:41:59.503]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.519]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:484b4aa6ae03433c epochNumber=1
{"provingJobId":"1:BASE_PARITY:484b4aa6ae03433c"}
16:41:59 [16:41:59.521]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9pzWIE/1 with map size 1073741824
16:41:59 [16:41:59.522]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.536]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce57372577bf2d86 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ce57372577bf2d86"}
16:41:59 [16:41:59.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:722d37f062e60e49 epochNumber=2
{"provingJobId":"1:BASE_PARITY:722d37f062e60e49"}
16:41:59 [16:41:59.542]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test9pzWIE/2 with map size 1073741824
16:41:59 [16:41:59.544]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.559]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99313b3b680475c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:99313b3b680475c5"}
16:41:59 [16:41:59.569]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.569]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:168f2e578ef3bfd6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:168f2e578ef3bfd6"}
16:41:59 [16:41:59.570]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDUPkc0/1 with map size 1073741824
16:41:59 [16:41:59.571]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.589]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:389cf1ef30604b2c epochNumber=1
{"provingJobId":"1:BASE_PARITY:389cf1ef30604b2c"}
16:41:59 [16:41:59.595]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17ab1c2732608b2b epochNumber=2
{"provingJobId":"1:BASE_PARITY:17ab1c2732608b2b"}
16:41:59 [16:41:59.596]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDUPkc0/2 with map size 1073741824
16:41:59 [16:41:59.599]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15cd022046e46c9d epochNumber=1
{"provingJobId":"1:BASE_PARITY:15cd022046e46c9d"}
16:41:59 [16:41:59.628]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.629]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e113d88911a1b6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:2e113d88911a1b6a"}
16:41:59 [16:41:59.630]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpJjOP6/1 with map size 1073741824
16:41:59 [16:41:59.632]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.648]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c12830ed4298206c epochNumber=1
{"provingJobId":"1:BASE_PARITY:c12830ed4298206c"}
16:41:59 [16:41:59.654]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88ff582d1c8f3459 epochNumber=2
{"provingJobId":"1:BASE_PARITY:88ff582d1c8f3459"}
16:41:59 [16:41:59.654]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpJjOP6/2 with map size 1073741824
16:41:59 [16:41:59.656]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.672]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7e4183cb9b7607f epochNumber=1
{"provingJobId":"1:BASE_PARITY:d7e4183cb9b7607f"}
16:41:59 [16:41:59.681]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.682]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:297f405b9a4c7b4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:297f405b9a4c7b4b"}
16:41:59 [16:41:59.683]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfHf9An/1 with map size 1073741824
16:41:59 [16:41:59.684]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44c9f52ff60493e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:44c9f52ff60493e9"}
16:41:59 [16:41:59.706]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:297f405b9a4c7b4b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:297f405b9a4c7b4b"}
16:41:59 [16:41:59.715]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.715]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10f8cd6fcaf0bf5b epochNumber=1
{"provingJobId":"1:BASE_PARITY:10f8cd6fcaf0bf5b"}
16:41:59 [16:41:59.716]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjf6x4e/1 with map size 1073741824
16:41:59 [16:41:59.717]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.733]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c7e99f4345010c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c7e99f4345010c7"}
16:41:59 [16:41:59.740]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:10f8cd6fcaf0bf5b type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:10f8cd6fcaf0bf5b"}
16:41:59 [16:41:59.748]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.749]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9eb73117543cffa epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9eb73117543cffa"}
16:41:59 [16:41:59.750]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3xISw0/1 with map size 1073741824
16:41:59 [16:41:59.751]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.767]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:db8a710bd98ca975 epochNumber=1
{"provingJobId":"1:BASE_PARITY:db8a710bd98ca975"}
16:41:59 [16:41:59.774]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b9eb73117543cffa type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:b9eb73117543cffa"}
16:41:59 [16:41:59.779]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.780]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:028c60bc875ca0a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:028c60bc875ca0a5"}
16:41:59 [16:41:59.780]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSGW4ik/1 with map size 1073741824
16:41:59 [16:41:59.782]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.799]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:028c60bc875ca0a5
{"provingJobId":"1:BASE_PARITY:028c60bc875ca0a5"}
16:41:59 [16:41:59.799]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:028c60bc875ca0a5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:028c60bc875ca0a5"}
16:41:59 [16:41:59.805]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56d8838a2ee7d861 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56d8838a2ee7d861"}
16:41:59 [16:41:59.811]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:028c60bc875ca0a5 has already been completed
{"provingJobId":"1:BASE_PARITY:028c60bc875ca0a5"}
16:41:59 [16:41:59.815]
INFO:
prover-client:proving-broker Proving Broker started
16:41:59 [16:41:59.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3275a0a82d535cab epochNumber=1
{"provingJobId":"1:BASE_PARITY:3275a0a82d535cab"}
16:41:59 [16:41:59.816]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8m4geS/1 with map size 1073741824
16:41:59 [16:41:59.818]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:41:59 [16:41:59.833]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f73f1d0c1bdc41a epochNumber=2
{"provingJobId":"1:BASE_PARITY:8f73f1d0c1bdc41a"}
16:41:59 [16:41:59.834]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8m4geS/2 with map size 1073741824
16:41:59 [16:41:59.836]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:00 [16:42:00.904]
INFO:
prover-client:proving-broker Proving Broker started
16:42:00 [16:42:00.915]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3275a0a82d535cab settled=false
{"provingJobId":"1:BASE_PARITY:3275a0a82d535cab","status":"pending"}
16:42:00 [16:42:00.917]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8f73f1d0c1bdc41a settled=false
{"provingJobId":"1:BASE_PARITY:8f73f1d0c1bdc41a","status":"pending"}
16:42:00 [16:42:00.918]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3275a0a82d535cab type=BASE_PARITY startedAt=1750092119852 older agent has taken job
{"provingJobId":"1:BASE_PARITY:3275a0a82d535cab"}
16:42:00 [16:42:00.918]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3275a0a82d535cab type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:3275a0a82d535cab"}
16:42:00 [16:42:00.921]
INFO:
prover-client:proving-broker Proving Broker started
16:42:00 [16:42:00.922]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6402807391c8403 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b6402807391c8403"}
16:42:00 [16:42:00.923]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testO0A7Zx/1 with map size 1073741824
16:42:00 [16:42:00.928]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:00 [16:42:00.947]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:008996326044b9b9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:008996326044b9b9"}
16:42:00 [16:42:00.948]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testO0A7Zx/2 with map size 1073741824
16:42:00 [16:42:00.950]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:01 [16:42:01.969]
INFO:
prover-client:proving-broker Proving Broker started
16:42:01 [16:42:01.971]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b6402807391c8403 settled=false
{"provingJobId":"1:BASE_PARITY:b6402807391c8403","status":"pending"}
16:42:01 [16:42:01.972]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:008996326044b9b9 settled=false
{"provingJobId":"1:BASE_PARITY:008996326044b9b9","status":"pending"}
16:42:01 [16:42:01.972]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b6402807391c8403 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:b6402807391c8403"}
16:42:01 [16:42:01.976]
INFO:
prover-client:proving-broker Proving Broker started
16:42:01 [16:42:01.977]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a14c1eae85df0f6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a14c1eae85df0f6a"}
16:42:01 [16:42:01.978]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testm9qzCW/1 with map size 1073741824
16:42:01 [16:42:01.979]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:02 [16:42:02.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ffe30bf34ee6fed epochNumber=2
{"provingJobId":"1:BASE_PARITY:7ffe30bf34ee6fed"}
16:42:02 [16:42:02.030]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testm9qzCW/2 with map size 1073741824
16:42:02 [16:42:02.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.040]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.043]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a14c1eae85df0f6a settled=false
{"provingJobId":"1:BASE_PARITY:a14c1eae85df0f6a","status":"pending"}
16:42:03 [16:42:03.044]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7ffe30bf34ee6fed settled=false
{"provingJobId":"1:BASE_PARITY:7ffe30bf34ee6fed","status":"pending"}
16:42:03 [16:42:03.045]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a14c1eae85df0f6a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a14c1eae85df0f6a"}
16:42:03 [16:42:03.046]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a14c1eae85df0f6a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a14c1eae85df0f6a"}
16:42:03 [16:42:03.056]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.057]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5068bc4c7915cc8c epochNumber=1
{"provingJobId":"1:BASE_PARITY:5068bc4c7915cc8c"}
16:42:03 [16:42:03.058]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyKerTD/1 with map size 1073741824
16:42:03 [16:42:03.060]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6297ea413fd6a68e epochNumber=2
{"provingJobId":"1:BASE_PARITY:6297ea413fd6a68e"}
16:42:03 [16:42:03.081]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyKerTD/2 with map size 1073741824
16:42:03 [16:42:03.083]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.100]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5068bc4c7915cc8c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5068bc4c7915cc8c"}
16:42:03 [16:42:03.109]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6297ea413fd6a68e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6297ea413fd6a68e"}
16:42:03 [16:42:03.121]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83116c291c54966d epochNumber=1
{"provingJobId":"1:BASE_PARITY:83116c291c54966d"}
16:42:03 [16:42:03.122]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuM9Djh/1 with map size 1073741824
16:42:03 [16:42:03.126]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.143]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1cd8d296e576ab7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e1cd8d296e576ab7"}
16:42:03 [16:42:03.143]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testuM9Djh/2 with map size 1073741824
16:42:03 [16:42:03.145]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.160]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:83116c291c54966d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:83116c291c54966d"}
16:42:03 [16:42:03.161]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:83116c291c54966d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:83116c291c54966d"}
16:42:03 [16:42:03.166]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e1cd8d296e576ab7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e1cd8d296e576ab7"}
16:42:03 [16:42:03.177]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.178]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:93767a4fa74bb76f err=test error
{"provingJoId":"1:BASE_PARITY:93767a4fa74bb76f"}
16:42:03 [16:42:03.181]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.182]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7a7de7775d576149 not found
{"provingJobId":"1:BASE_PARITY:7a7de7775d576149"}
16:42:03 [16:42:03.186]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.187]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98ba1c204e99f425 epochNumber=1
{"provingJobId":"1:BASE_PARITY:98ba1c204e99f425"}
16:42:03 [16:42:03.188]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYom4qL/1 with map size 1073741824
16:42:03 [16:42:03.189]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.213]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.218]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8007a81975849244 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8007a81975849244"}
16:42:03 [16:42:03.219]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdXVJKl/1 with map size 1073741824
16:42:03 [16:42:03.225]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.342]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8007a81975849244 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8007a81975849244"}
16:42:03 [16:42:03.350]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.350]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da785aab8a24ccfc epochNumber=1
{"provingJobId":"1:BASE_PARITY:da785aab8a24ccfc"}
16:42:03 [16:42:03.351]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9qHPYI/1 with map size 1073741824
16:42:03 [16:42:03.352]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.479]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:da785aab8a24ccfc timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:da785aab8a24ccfc"}
16:42:03 [16:42:03.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80c31eb326950f1e epochNumber=10
{"provingJobId":"1:BASE_PARITY:80c31eb326950f1e"}
16:42:03 [16:42:03.485]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test9qHPYI/10 with map size 1073741824
16:42:03 [16:42:03.487]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.531]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:42:03 [16:42:03.531]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:42:03 [16:42:03.644]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.645]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:befd0e64d825e0e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:befd0e64d825e0e5"}
16:42:03 [16:42:03.646]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUijs5s/1 with map size 1073741824
16:42:03 [16:42:03.647]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.822]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:befd0e64d825e0e5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:befd0e64d825e0e5"}
16:42:03 [16:42:03.871]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:311525a345948f92 epochNumber=1
{"provingJobId":"1:BASE_PARITY:311525a345948f92"}
16:42:03 [16:42:03.873]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnAwlnh/1 with map size 1073741824
16:42:03 [16:42:03.875]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.896]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:311525a345948f92 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:311525a345948f92"}
16:42:03 [16:42:03.904]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.905]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6262a29bdb28a48 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6262a29bdb28a48"}
16:42:03 [16:42:03.906]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testspQthf/1 with map size 1073741824
16:42:03 [16:42:03.908]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.937]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a6262a29bdb28a48 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a6262a29bdb28a48"}
16:42:03 [16:42:03.937]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a6262a29bdb28a48 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:a6262a29bdb28a48"}
16:42:03 [16:42:03.948]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.949]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f36ff2f14556608 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f36ff2f14556608"}
16:42:03 [16:42:03.949]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvsWrVk/1 with map size 1073741824
16:42:03 [16:42:03.951]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:03 [16:42:03.971]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2f36ff2f14556608 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2f36ff2f14556608"}
16:42:03 [16:42:03.989]
INFO:
prover-client:proving-broker Proving Broker started
16:42:03 [16:42:03.990]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:249be0df5feb4116 epochNumber=1
{"provingJobId":"1:BASE_PARITY:249be0df5feb4116"}
16:42:03 [16:42:03.991]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWF9C0U/1 with map size 1073741824
16:42:03 [16:42:03.995]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.028]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:249be0df5feb4116 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:249be0df5feb4116"}
16:42:04 [16:42:04.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42028f3d8fc180b9 epochNumber=3
{"provingJobId":"1:BASE_PARITY:42028f3d8fc180b9"}
16:42:04 [16:42:04.031]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testWF9C0U/3 with map size 1073741824
16:42:04 [16:42:04.037]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.066]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:42:04 [16:42:04.066]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:42:04 [16:42:04.082]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:249be0df5feb4116 err=test error
{"provingJoId":"1:BASE_PARITY:249be0df5feb4116"}
16:42:04 [16:42:04.090]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOi5P81/1 with map size 1073741824
16:42:04 [16:42:04.095]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.129]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOi5P81/2 with map size 1073741824
16:42:04 [16:42:04.133]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.150]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.154]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:29b23637ab5c1f85 settled=false
{"provingJobId":"1:BASE_PARITY:29b23637ab5c1f85","status":"pending"}
16:42:04 [16:42:04.157]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:393d83749fc8e4b0 settled=false
{"provingJobId":"1:BASE_PARITY:393d83749fc8e4b0","status":"pending"}
16:42:04 [16:42:04.165]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfG6xRL/1 with map size 1073741824
16:42:04 [16:42:04.166]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.190]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfG6xRL/2 with map size 1073741824
16:42:04 [16:42:04.194]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.226]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.228]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0996fd471dccb1fd settled=true
{"provingJobId":"1:BASE_PARITY:0996fd471dccb1fd","status":"fulfilled"}
16:42:04 [16:42:04.230]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:124a53422a947406 settled=true
{"provingJobId":"2:BASE_PARITY:124a53422a947406","status":"fulfilled"}
16:42:04 [16:42:04.237]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFitWoW/1 with map size 1073741824
16:42:04 [16:42:04.239]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.268]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFitWoW/2 with map size 1073741824
16:42:04 [16:42:04.270]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.286]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.288]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e05530ef38f7eb11 settled=true
{"provingJobId":"1:BASE_PARITY:e05530ef38f7eb11","status":"fulfilled"}
16:42:04 [16:42:04.290]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a3f4730b520be8b4 settled=false
{"provingJobId":"1:BASE_PARITY:a3f4730b520be8b4","status":"pending"}
16:42:04 [16:42:04.295]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.296]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e49df6d3cd0ba5a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e49df6d3cd0ba5a6"}
16:42:04 [16:42:04.297]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgYMNHm/1 with map size 1073741824
16:42:04 [16:42:04.298]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.319]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.319]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0ec0ddfdc294f6f epochNumber=1
{"provingJobId":"1:BASE_PARITY:f0ec0ddfdc294f6f"}
16:42:04 [16:42:04.320]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:f0ec0ddfdc294f6f: Error: db error: Error: db error
16:42:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:f0ec0ddfdc294f6f"}
16:42:04 [16:42:04.324]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.324]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:30f61a3fcad02861 epochNumber=1
{"provingJobId":"1:BASE_PARITY:30f61a3fcad02861"}
16:42:04 [16:42:04.325]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuptZtz/1 with map size 1073741824
16:42:04 [16:42:04.326]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.342]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:30f61a3fcad02861 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:30f61a3fcad02861"}
16:42:04 [16:42:04.342]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:30f61a3fcad02861 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:30f61a3fcad02861"}
16:42:04 [16:42:04.351]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.352]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3bf3f0dacc706c14 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3bf3f0dacc706c14"}
16:42:04 [16:42:04.353]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPcaJoR/1 with map size 1073741824
16:42:04 [16:42:04.354]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.370]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3bf3f0dacc706c14 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3bf3f0dacc706c14"}
16:42:04 [16:42:04.371]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3bf3f0dacc706c14 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3bf3f0dacc706c14"}
16:42:04 [16:42:04.372]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:3bf3f0dacc706c14: Error: db error
16:42:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:3bf3f0dacc706c14"}
16:42:04 [16:42:04.376]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.377]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:427fe3c56d74f87f epochNumber=1
{"provingJobId":"1:BASE_PARITY:427fe3c56d74f87f"}
16:42:04 [16:42:04.379]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testh4yAj8/1 with map size 1073741824
16:42:04 [16:42:04.381]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.398]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:427fe3c56d74f87f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:427fe3c56d74f87f"}
16:42:04 [16:42:04.399]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:427fe3c56d74f87f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:427fe3c56d74f87f"}
16:42:04 [16:42:04.411]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.412]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43d634166d683726 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43d634166d683726"}
16:42:04 [16:42:04.413]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4CPWmq/1 with map size 1073741824
16:42:04 [16:42:04.415]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.432]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:43d634166d683726 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:43d634166d683726"}
16:42:04 [16:42:04.432]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:43d634166d683726 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:43d634166d683726"}
16:42:04 [16:42:04.433]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:43d634166d683726 jobErr=test error: Error: db error
16:42:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:43d634166d683726"}
16:42:04 [16:42:04.438]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.438]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2a166acb82ba1905 not found
{"provingJobId":"1:BASE_PARITY:2a166acb82ba1905"}
16:42:04 [16:42:04.442]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.443]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:95838c6ce3cc6a3e err=test error
{"provingJoId":"1:BASE_PARITY:95838c6ce3cc6a3e"}
16:42:04 [16:42:04.447]
INFO:
prover-client:proving-broker Proving Broker started
16:42:04 [16:42:04.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b1dd18cfa11f5964 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b1dd18cfa11f5964"}
16:42:04 [16:42:04.460]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCuUcOZ/1 with map size 1073741824
16:42:04 [16:42:04.463]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.480]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b1dd18cfa11f5964 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b1dd18cfa11f5964"}
16:42:04 [16:42:04.480]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b1dd18cfa11f5964 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b1dd18cfa11f5964"}
16:42:04 [16:42:04.497]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:302ddfa04a5d2c48 epochNumber=2
{"provingJobId":"2:BASE_PARITY:302ddfa04a5d2c48"}
16:42:04 [16:42:04.499]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCuUcOZ/2 with map size 1073741824
16:42:04 [16:42:04.501]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.522]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:302ddfa04a5d2c48 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:302ddfa04a5d2c48"}
16:42:04 [16:42:04.522]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:302ddfa04a5d2c48 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:302ddfa04a5d2c48"}
16:42:04 [16:42:04.590]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:93544ecda79a29b3 epochNumber=3
{"provingJobId":"3:BASE_PARITY:93544ecda79a29b3"}
16:42:04 [16:42:04.592]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testCuUcOZ/3 with map size 1073741824
16:42:04 [16:42:04.595]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.627]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:42:04 [16:42:04.628]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:42:04 [16:42:04.676]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:7ec876fcd2c8251b epochNumber=4
{"provingJobId":"4:BASE_PARITY:7ec876fcd2c8251b"}
16:42:04 [16:42:04.677]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testCuUcOZ/4 with map size 1073741824
16:42:04 [16:42:04.681]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.706]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:42:04 [16:42:04.706]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
16:42:04 [16:42:04.761]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:cd8adca22743f0f5 epochNumber=5
{"provingJobId":"5:BASE_PARITY:cd8adca22743f0f5"}
16:42:04 [16:42:04.762]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testCuUcOZ/5 with map size 1073741824
16:42:04 [16:42:04.765]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:42:04 [16:42:04.783]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:42:04 [16:42:04.784]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
16:42:04
PASS src/proving_broker/proving_broker.test.ts (
13.245 s)
16:42:04 ProvingBroker
16:42:04 Producer API
16:42:04
✓ refuses stale jobs (41 ms)
16:42:04
✓ enqueues jobs (4 ms)
16:42:04
✓ ignores duplicate jobs (4 ms)
16:42:04
✓ reports correct status when enqueuing repeat jobs (4 ms)
16:42:04
✓ throws an error in case of duplicate job IDs (28 ms)
16:42:04
✓ returns not-found status for non-existing jobs (2 ms)
16:42:04
✓ cancels jobs in queue (3 ms)
16:42:04
✓ cancels jobs in-progress (2 ms)
16:42:04
✓ returns job result if successful (3 ms)
16:42:04
✓ returns job error if failed (2 ms)
16:42:04
✓ correctly returns job status for concurrent writes (2 ms)
16:42:04
✓ refuses stale jobs (44 ms)
16:42:04
✓ enqueues jobs (34 ms)
16:42:04
✓ ignores duplicate jobs (25 ms)
16:42:04
✓ reports correct status when enqueuing repeat jobs (36 ms)
16:42:04
✓ throws an error in case of duplicate job IDs (26 ms)
16:42:04
✓ returns not-found status for non-existing jobs (2 ms)
16:42:04
✓ cancels jobs in queue (36 ms)
16:42:04
✓ cancels jobs in-progress (29 ms)
16:42:04
✓ returns job result if successful (31 ms)
16:42:04
✓ returns job error if failed (36 ms)
16:42:04
✓ correctly returns job status for concurrent writes (40 ms)
16:42:04 Consumer API
16:42:04
✓ returns undefined if no jobs are available (1 ms)
16:42:04
✓ returns jobs in priority order (2 ms)
16:42:04
✓ returns undefined if no jobs are available for the given allowList (1 ms)
16:42:04
✓ returns a job if it is in the allowList (2 ms)
16:42:04
✓ returns the most important job if it is in the allowList (2 ms)
16:42:04
✓ returns any job if filter is empty (2 ms)
16:42:04
✓ returns a new job when reporting job success (3 ms)
16:42:04
✓ returns a new job when reporting permanent error (3 ms)
16:42:04
✓ returns a new job when reporting retry-able error (2 ms)
16:42:04
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
16:42:04
✓ returns a new job if job is already in progress elsewhere (1063 ms)
16:42:04
✓ avoids sending the same job to a new agent after a restart (1007 ms)
16:42:04
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
16:42:04
✓ tracks job result if in progress (2 ms)
16:42:04
✓ tracks job result even if job is in queue (3 ms)
16:42:04
✓ ignores reported job error if unknown job (2 ms)
16:42:04
✓ ignores job result if unknown job (1 ms)
16:42:04
✓ returns undefined if no jobs are available (2 ms)
16:42:04
✓ returns jobs in priority order (28 ms)
16:42:04
✓ returns undefined if no jobs are available for the given allowList (20 ms)
16:42:04
✓ returns a job if it is in the allowList (49 ms)
16:42:04
✓ returns the most important job if it is in the allowList (59 ms)
16:42:04
✓ returns any job if filter is empty (53 ms)
16:42:04
✓ returns a new job when reporting job success (33 ms)
16:42:04
✓ returns a new job when reporting permanent error (33 ms)
16:42:04
✓ returns a new job when reporting retry-able error (30 ms)
16:42:04
✓ returns a new job when reporting progress if current one is cancelled (36 ms)
16:42:04
✓ returns a new job if job is already in progress elsewhere (1107 ms)
16:42:04
✓ avoids sending the same job to a new agent after a restart (1054 ms)
16:42:04
✓ avoids sending a completed job to a new agent after a restart (1078 ms)
16:42:04
✓ tracks job result if in progress (64 ms)
16:42:04
✓ tracks job result even if job is in queue (55 ms)
16:42:04
✓ ignores reported job error if unknown job (4 ms)
16:42:04
✓ ignores job result if unknown job (4 ms)
16:42:04 Timeouts
16:42:04
✓ tracks in progress jobs (2 ms)
16:42:04
✓ re-enqueues jobs that time out (129 ms)
16:42:04
✓ cancel stale jobs that time out (236 ms)
16:42:04
✓ keeps the jobs in progress while it is alive (209 ms)
16:42:04
✓ tracks in progress jobs (26 ms)
16:42:04
✓ re-enqueues jobs that time out (138 ms)
16:42:04
✓ cancel stale jobs that time out (293 ms)
16:42:04
✓ keeps the jobs in progress while it is alive (227 ms)
16:42:04 Retries
16:42:04
✓ retries jobs (2 ms)
16:42:04
✓ retries up to a maximum number of times (2 ms)
16:42:04
✓ passing retry=false does not retry (2 ms)
16:42:04
✓ does not retry if job is stale (29 ms)
16:42:04
✓ retries jobs (28 ms)
16:42:04
✓ retries up to a maximum number of times (47 ms)
16:42:04
✓ passing retry=false does not retry (38 ms)
16:42:04
✓ does not retry if job is stale (98 ms)
16:42:04 Database management
16:42:04
✓ re-enqueues proof requests on start (2 ms)
16:42:04
✓ restores proof results on start (2 ms)
16:42:04
✓ only re-enqueues unfinished jobs (2 ms)
16:42:04
✓ saves job when enqueued (2 ms)
16:42:04
✓ does not retain job if database fails to save (2 ms)
16:42:04
✓ saves job result (2 ms)
16:42:04
✓ saves result even if database fails to save (2 ms)
16:42:04
✓ saves job error (1 ms)
16:42:04
✓ saves job error even if database fails to save (2 ms)
16:42:04
✓ does not save job result if job is unknown (1 ms)
16:42:04
✓ does not save job error if job is unknown (1 ms)
16:42:04
✓ cleans up old jobs periodically (261 ms)
16:42:04
✓ re-enqueues proof requests on start (77 ms)
16:42:04
✓ restores proof results on start (72 ms)
16:42:04
✓ only re-enqueues unfinished jobs (59 ms)
16:42:04
✓ saves job when enqueued (22 ms)
16:42:04
✓ does not retain job if database fails to save (6 ms)
16:42:04
✓ saves job result (27 ms)
16:42:04
✓ saves result even if database fails to save (25 ms)
16:42:04
✓ saves job error (33 ms)
16:42:04
✓ saves job error even if database fails to save (27 ms)
16:42:04
✓ does not save job result if job is unknown (4 ms)
16:42:04
✓ does not save job error if job is unknown (4 ms)
16:42:04
✓ cleans up old jobs periodically (393 ms)
16:42:04
16:42:04
Test Suites: 1 passed, 1 total
16:42:04
Tests: 96 passed, 96 total
16:42:04
Snapshots: 0 total
16:42:04
Time: 13.379 s
16:42:04
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
16:42:04
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?