Parent Log:
http://ci.aztec-labs.com/ee44175121670d05
Command: 5484f2347094f66b yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/5ca43853c40a2b8bfbc2148d5fdd46c6f82a74eb
Env: REF_NAME=gh-readonly-queue/next/pr-14178-3cbe7945c84145b8646cf5cf8fb730b915c0dc93 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 16 13:32:56 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14178_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
13:33:00 [13:33:00.968]
INFO:
prover-client:proving-broker Proving Broker started
13:33:00 [13:33:00.977]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a199a4c7b740ddba epochNumber=42
{"provingJobId":"1:BASE_PARITY:a199a4c7b740ddba"}
13:33:00 [13:33:00.981]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:1908efa07a13d76f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1908efa07a13d76f"}
13:33:01 [13:33:01.005]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.006]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d0793acdab0ccceb epochNumber=1
{"provingJobId":"1:BASE_PARITY:d0793acdab0ccceb"}
13:33:01 [13:33:01.007]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92323eabf7199d1d epochNumber=1
{"provingJobId":"1:BASE_PARITY:92323eabf7199d1d"}
13:33:01 [13:33:01.008]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.009]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54b9319ce5488a79 epochNumber=1
{"provingJobId":"1:BASE_PARITY:54b9319ce5488a79"}
13:33:01 [13:33:01.010]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:54b9319ce5488a79 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:54b9319ce5488a79"}
13:33:01 [13:33:01.012]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6be7f0b1e6bc220c epochNumber=1
{"provingJobId":"1:BASE_PARITY:6be7f0b1e6bc220c"}
13:33:01 [13:33:01.013]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6be7f0b1e6bc220c epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6be7f0b1e6bc220c"}
13:33:01 [13:33:01.014]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6be7f0b1e6bc220c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6be7f0b1e6bc220c"}
13:33:01 [13:33:01.014]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6be7f0b1e6bc220c epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6be7f0b1e6bc220c"}
13:33:01 [13:33:01.015]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.016]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28464f706fd7815c epochNumber=1
{"provingJobId":"1:BASE_PARITY:28464f706fd7815c"}
13:33:01 [13:33:01.035]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.046]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.047]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df514cbad711bbd4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df514cbad711bbd4"}
13:33:01 [13:33:01.047]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:df514cbad711bbd4
{"provingJobId":"1:BASE_PARITY:df514cbad711bbd4"}
13:33:01 [13:33:01.048]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:df514cbad711bbd4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:df514cbad711bbd4"}
13:33:01 [13:33:01.048]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:df514cbad711bbd4 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:df514cbad711bbd4"}
13:33:01 [13:33:01.049]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.050]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3fefafb59c64d577 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3fefafb59c64d577"}
13:33:01 [13:33:01.050]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3fefafb59c64d577
{"provingJobId":"1:BASE_PARITY:3fefafb59c64d577"}
13:33:01 [13:33:01.051]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3fefafb59c64d577 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3fefafb59c64d577"}
13:33:01 [13:33:01.053]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.053]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:975780721d933e7e epochNumber=1
{"provingJobId":"1:BASE_PARITY:975780721d933e7e"}
13:33:01 [13:33:01.054]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:975780721d933e7e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:975780721d933e7e"}
13:33:01 [13:33:01.054]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:975780721d933e7e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:975780721d933e7e"}
13:33:01 [13:33:01.061]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.062]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c28f45227d7c4457 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c28f45227d7c4457"}
13:33:01 [13:33:01.063]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c28f45227d7c4457 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c28f45227d7c4457"}
13:33:01 [13:33:01.063]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c28f45227d7c4457 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c28f45227d7c4457"}
13:33:01 [13:33:01.065]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.066]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfcf2c6077e94ab8 epochNumber=0
{"provingJobId":"1:BASE_PARITY:dfcf2c6077e94ab8"}
13:33:01 [13:33:01.066]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dfcf2c6077e94ab8 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dfcf2c6077e94ab8"}
13:33:01 [13:33:01.067]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d135b76b65994e4a epochNumber=0
{"provingJobId":"1:BASE_PARITY:d135b76b65994e4a"}
13:33:01 [13:33:01.067]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dfcf2c6077e94ab8 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dfcf2c6077e94ab8"}
13:33:01 [13:33:01.067]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99dd01b4a0a4233b epochNumber=0
{"provingJobId":"1:BASE_PARITY:99dd01b4a0a4233b"}
13:33:01 [13:33:01.069]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.070]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.071]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50f6c02f8d05614e epochNumber=2
{"provingJobId":"1:BASE_PARITY:50f6c02f8d05614e"}
13:33:01 [13:33:01.071]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:029e41627a509d6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:029e41627a509d6a"}
13:33:01 [13:33:01.073]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0c89e9399f27409 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f0c89e9399f27409"}
13:33:01 [13:33:01.075]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.076]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7503d9f71bef3239 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7503d9f71bef3239"}
13:33:01 [13:33:01.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:adf1c5459db0724d epochNumber=1
{"provingJobId":"1:BASE_PARITY:adf1c5459db0724d"}
13:33:01 [13:33:01.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f6471bdf6d0cae8b epochNumber=2
{"provingJobId":"1:BASE_PARITY:f6471bdf6d0cae8b"}
13:33:01 [13:33:01.078]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e5e1cc432705654 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e5e1cc432705654"}
13:33:01 [13:33:01.079]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.080]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32aa4b7e17f0204b epochNumber=1
{"provingJobId":"1:BASE_PARITY:32aa4b7e17f0204b"}
13:33:01 [13:33:01.080]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af3ba722100f69a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af3ba722100f69a5"}
13:33:01 [13:33:01.081]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9efe3694f4a6756c epochNumber=2
{"provingJobId":"1:BASE_PARITY:9efe3694f4a6756c"}
13:33:01 [13:33:01.081]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ba55075c0a59eff epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ba55075c0a59eff"}
13:33:01 [13:33:01.082]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b048f4754a10384e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b048f4754a10384e"}
13:33:01 [13:33:01.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d64d9589acde0afd epochNumber=1
{"provingJobId":"1:BASE_PARITY:d64d9589acde0afd"}
13:33:01 [13:33:01.083]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff8d0697e99a984a epochNumber=2
{"provingJobId":"1:BASE_PARITY:ff8d0697e99a984a"}
13:33:01 [13:33:01.083]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55abfddf552f43a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:55abfddf552f43a1"}
13:33:01 [13:33:01.084]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.084]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1202de3b58d7d6fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:1202de3b58d7d6fb"}
13:33:01 [13:33:01.084]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4550bcbc32e87626 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4550bcbc32e87626"}
13:33:01 [13:33:01.085]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1202de3b58d7d6fb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1202de3b58d7d6fb"}
13:33:01 [13:33:01.085]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5766af1f9dfdf7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:f5766af1f9dfdf7d"}
13:33:01 [13:33:01.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa72797210cd071f epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa72797210cd071f"}
13:33:01 [13:33:01.086]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f5766af1f9dfdf7d type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:f5766af1f9dfdf7d"}
13:33:01 [13:33:01.087]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f28080ce9c4a38a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f28080ce9c4a38a8"}
13:33:01 [13:33:01.088]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e92661ad7f5d3b63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e92661ad7f5d3b63"}
13:33:01 [13:33:01.088]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f28080ce9c4a38a8 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:f28080ce9c4a38a8"}
13:33:01 [13:33:01.089]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.089]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45987728e19df13a epochNumber=1
{"provingJobId":"1:BASE_PARITY:45987728e19df13a"}
13:33:01 [13:33:01.089]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:45987728e19df13a
{"provingJobId":"1:BASE_PARITY:45987728e19df13a"}
13:33:01 [13:33:01.090]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:45987728e19df13a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:45987728e19df13a"}
13:33:01 [13:33:01.090]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3652bf62ad3baff6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3652bf62ad3baff6"}
13:33:01 [13:33:01.090]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:45987728e19df13a has already been completed
{"provingJobId":"1:BASE_PARITY:45987728e19df13a"}
13:33:01 [13:33:01.091]
INFO:
prover-client:proving-broker Proving Broker started
13:33:01 [13:33:01.091]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ed3aa43c87619dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ed3aa43c87619dc"}
13:33:01 [13:33:01.091]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ea4190dc1b1f15c epochNumber=2
{"provingJobId":"1:BASE_PARITY:3ea4190dc1b1f15c"}
13:33:02 [13:33:02.143]
INFO:
prover-client:proving-broker Proving Broker started
13:33:02 [13:33:02.144]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4ed3aa43c87619dc settled=false
{"provingJobId":"1:BASE_PARITY:4ed3aa43c87619dc","status":"pending"}
13:33:02 [13:33:02.144]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3ea4190dc1b1f15c settled=false
{"provingJobId":"1:BASE_PARITY:3ea4190dc1b1f15c","status":"pending"}
13:33:02 [13:33:02.145]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4ed3aa43c87619dc type=BASE_PARITY startedAt=1750080781092 older agent has taken job
{"provingJobId":"1:BASE_PARITY:4ed3aa43c87619dc"}
13:33:02 [13:33:02.146]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4ed3aa43c87619dc type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:4ed3aa43c87619dc"}
13:33:02 [13:33:02.147]
INFO:
prover-client:proving-broker Proving Broker started
13:33:02 [13:33:02.147]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bdb7046f1d5e343b epochNumber=1
{"provingJobId":"1:BASE_PARITY:bdb7046f1d5e343b"}
13:33:02 [13:33:02.148]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb851d4ee4a7ae0f epochNumber=2
{"provingJobId":"1:BASE_PARITY:fb851d4ee4a7ae0f"}
13:33:03 [13:33:03.149]
INFO:
prover-client:proving-broker Proving Broker started
13:33:03 [13:33:03.149]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bdb7046f1d5e343b settled=false
{"provingJobId":"1:BASE_PARITY:bdb7046f1d5e343b","status":"pending"}
13:33:03 [13:33:03.150]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fb851d4ee4a7ae0f settled=false
{"provingJobId":"1:BASE_PARITY:fb851d4ee4a7ae0f","status":"pending"}
13:33:03 [13:33:03.150]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bdb7046f1d5e343b type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:bdb7046f1d5e343b"}
13:33:03 [13:33:03.152]
INFO:
prover-client:proving-broker Proving Broker started
13:33:03 [13:33:03.153]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9d30dc19919d238 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c9d30dc19919d238"}
13:33:03 [13:33:03.153]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2ff465d9785c928 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f2ff465d9785c928"}
13:33:04 [13:33:04.154]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.154]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c9d30dc19919d238 settled=false
{"provingJobId":"1:BASE_PARITY:c9d30dc19919d238","status":"pending"}
13:33:04 [13:33:04.154]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f2ff465d9785c928 settled=false
{"provingJobId":"1:BASE_PARITY:f2ff465d9785c928","status":"pending"}
13:33:04 [13:33:04.155]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c9d30dc19919d238 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c9d30dc19919d238"}
13:33:04 [13:33:04.155]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c9d30dc19919d238 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c9d30dc19919d238"}
13:33:04 [13:33:04.158]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:637a7cec7d2fe354 epochNumber=1
{"provingJobId":"1:BASE_PARITY:637a7cec7d2fe354"}
13:33:04 [13:33:04.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25a85100ba8f919a epochNumber=2
{"provingJobId":"1:BASE_PARITY:25a85100ba8f919a"}
13:33:04 [13:33:04.160]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:637a7cec7d2fe354 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:637a7cec7d2fe354"}
13:33:04 [13:33:04.160]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:25a85100ba8f919a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:25a85100ba8f919a"}
13:33:04 [13:33:04.169]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.175]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16aa48e98cb5df3d epochNumber=1
{"provingJobId":"1:BASE_PARITY:16aa48e98cb5df3d"}
13:33:04 [13:33:04.175]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cae1ae01643f84d0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cae1ae01643f84d0"}
13:33:04 [13:33:04.176]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:16aa48e98cb5df3d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:16aa48e98cb5df3d"}
13:33:04 [13:33:04.176]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:16aa48e98cb5df3d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:16aa48e98cb5df3d"}
13:33:04 [13:33:04.178]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cae1ae01643f84d0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:cae1ae01643f84d0"}
13:33:04 [13:33:04.179]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.180]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4e003c8d3efa6356 err=test error
{"provingJoId":"1:BASE_PARITY:4e003c8d3efa6356"}
13:33:04 [13:33:04.181]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.182]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3c0f9575bec6a9ed not found
{"provingJobId":"1:BASE_PARITY:3c0f9575bec6a9ed"}
13:33:04 [13:33:04.183]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.184]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c28f1c3b7fd70fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c28f1c3b7fd70fb"}
13:33:04 [13:33:04.185]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.186]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48a9e7f46575b491 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48a9e7f46575b491"}
13:33:04 [13:33:04.286]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:48a9e7f46575b491 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:48a9e7f46575b491"}
13:33:04 [13:33:04.289]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.290]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c27c24de7c9b6c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c27c24de7c9b6c5"}
13:33:04 [13:33:04.393]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1c27c24de7c9b6c5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1c27c24de7c9b6c5"}
13:33:04 [13:33:04.400]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ad646ec1f3814c8 epochNumber=10
{"provingJobId":"1:BASE_PARITY:3ad646ec1f3814c8"}
13:33:04 [13:33:04.419]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:04 [13:33:04.527]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfae2b91b80bbbcc epochNumber=1
{"provingJobId":"1:BASE_PARITY:dfae2b91b80bbbcc"}
13:33:04 [13:33:04.702]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dfae2b91b80bbbcc timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:dfae2b91b80bbbcc"}
13:33:04 [13:33:04.733]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.733]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb767bdaad488254 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb767bdaad488254"}
13:33:04 [13:33:04.734]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fb767bdaad488254 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:fb767bdaad488254"}
13:33:04 [13:33:04.735]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.735]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd1412c151f6a747 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd1412c151f6a747"}
13:33:04 [13:33:04.736]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bd1412c151f6a747 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:bd1412c151f6a747"}
13:33:04 [13:33:04.736]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bd1412c151f6a747 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:bd1412c151f6a747"}
13:33:04 [13:33:04.737]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.738]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7a9052044a8ec4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d7a9052044a8ec4b"}
13:33:04 [13:33:04.738]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d7a9052044a8ec4b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d7a9052044a8ec4b"}
13:33:04 [13:33:04.739]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.739]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79a04adbe9882ffa epochNumber=1
{"provingJobId":"1:BASE_PARITY:79a04adbe9882ffa"}
13:33:04 [13:33:04.740]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:79a04adbe9882ffa type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:79a04adbe9882ffa"}
13:33:04 [13:33:04.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2cc07923b62a7f3b epochNumber=3
{"provingJobId":"1:BASE_PARITY:2cc07923b62a7f3b"}
13:33:04 [13:33:04.765]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:04 [13:33:04.766]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:79a04adbe9882ffa err=test error
{"provingJoId":"1:BASE_PARITY:79a04adbe9882ffa"}
13:33:04 [13:33:04.768]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.768]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:abd7480dbb4a46f0 settled=false
{"provingJobId":"1:BASE_PARITY:abd7480dbb4a46f0","status":"pending"}
13:33:04 [13:33:04.768]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:aad4791ec2ed0f33 settled=false
{"provingJobId":"1:BASE_PARITY:aad4791ec2ed0f33","status":"pending"}
13:33:04 [13:33:04.772]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.773]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:06ec63caf167e86f settled=true
{"provingJobId":"1:BASE_PARITY:06ec63caf167e86f","status":"fulfilled"}
13:33:04 [13:33:04.774]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b9958edfeaedc75c settled=true
{"provingJobId":"2:BASE_PARITY:b9958edfeaedc75c","status":"fulfilled"}
13:33:04 [13:33:04.775]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.776]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2554d615ac422f10 settled=true
{"provingJobId":"1:BASE_PARITY:2554d615ac422f10","status":"fulfilled"}
13:33:04 [13:33:04.776]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c452f2282286e769 settled=false
{"provingJobId":"1:BASE_PARITY:c452f2282286e769","status":"pending"}
13:33:04 [13:33:04.778]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.780]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:685a55aa6bae9dc0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:685a55aa6bae9dc0"}
13:33:04 [13:33:04.782]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b423502617b27b6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:b423502617b27b6a"}
13:33:04 [13:33:04.784]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:b423502617b27b6a: Error: db error: Error: db error
13:33:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:b423502617b27b6a"}
13:33:04 [13:33:04.785]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66459fd18c78d191 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66459fd18c78d191"}
13:33:04 [13:33:04.785]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:66459fd18c78d191 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:66459fd18c78d191"}
13:33:04 [13:33:04.786]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:66459fd18c78d191 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:66459fd18c78d191"}
13:33:04 [13:33:04.787]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.787]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:27f98a4c96b4fdb2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:27f98a4c96b4fdb2"}
13:33:04 [13:33:04.787]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:27f98a4c96b4fdb2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:27f98a4c96b4fdb2"}
13:33:04 [13:33:04.788]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:27f98a4c96b4fdb2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:27f98a4c96b4fdb2"}
13:33:04 [13:33:04.788]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:27f98a4c96b4fdb2: Error: db error
13:33:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:27f98a4c96b4fdb2"}
13:33:04 [13:33:04.789]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.789]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3954f704c3a17de0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3954f704c3a17de0"}
13:33:04 [13:33:04.790]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3954f704c3a17de0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3954f704c3a17de0"}
13:33:04 [13:33:04.790]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3954f704c3a17de0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3954f704c3a17de0"}
13:33:04 [13:33:04.791]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7e46e18d50df48d epochNumber=1
{"provingJobId":"1:BASE_PARITY:d7e46e18d50df48d"}
13:33:04 [13:33:04.792]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d7e46e18d50df48d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d7e46e18d50df48d"}
13:33:04 [13:33:04.792]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d7e46e18d50df48d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d7e46e18d50df48d"}
13:33:04 [13:33:04.792]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:d7e46e18d50df48d jobErr=test error: Error: db error
13:33:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:d7e46e18d50df48d"}
13:33:04 [13:33:04.794]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.794]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:96b03e06c30adf1f not found
{"provingJobId":"1:BASE_PARITY:96b03e06c30adf1f"}
13:33:04 [13:33:04.795]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.796]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:478a3f277a879cc9 err=test error
{"provingJoId":"1:BASE_PARITY:478a3f277a879cc9"}
13:33:04 [13:33:04.797]
INFO:
prover-client:proving-broker Proving Broker started
13:33:04 [13:33:04.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15c56a0c22575165 epochNumber=1
{"provingJobId":"1:BASE_PARITY:15c56a0c22575165"}
13:33:04 [13:33:04.807]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:15c56a0c22575165 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:15c56a0c22575165"}
13:33:04 [13:33:04.807]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:15c56a0c22575165 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:15c56a0c22575165"}
13:33:04 [13:33:04.818]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:83b49903bdf9723f epochNumber=2
{"provingJobId":"2:BASE_PARITY:83b49903bdf9723f"}
13:33:04 [13:33:04.819]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:83b49903bdf9723f type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:83b49903bdf9723f"}
13:33:04 [13:33:04.819]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:83b49903bdf9723f type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:83b49903bdf9723f"}
13:33:04 [13:33:04.881]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:9489ddaa390b14f4 epochNumber=3
{"provingJobId":"3:BASE_PARITY:9489ddaa390b14f4"}
13:33:04 [13:33:04.899]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:04 [13:33:04.943]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:125e7e1455bfc1a0 epochNumber=4
{"provingJobId":"4:BASE_PARITY:125e7e1455bfc1a0"}
13:33:04 [13:33:04.950]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:05 [13:33:05.005]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:75331eb8d29562a8 epochNumber=5
{"provingJobId":"5:BASE_PARITY:75331eb8d29562a8"}
13:33:05 [13:33:05.028]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:05 [13:33:05.063]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.064]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:add89713f0e4b920 epochNumber=42
{"provingJobId":"1:BASE_PARITY:add89713f0e4b920"}
13:33:05 [13:33:05.069]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testp85zhF/42 with map size 1073741824
13:33:05 [13:33:05.082]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.110]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:b272a332761d94fc epochNumber=1
{"provingJobId":"1:BASE_PARITY:b272a332761d94fc"}
13:33:05 [13:33:05.117]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.117]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd1163b1ebaa5723 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd1163b1ebaa5723"}
13:33:05 [13:33:05.121]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testp1Z8kG/1 with map size 1073741824
13:33:05 [13:33:05.123]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.147]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:432e8f4d8b2a2404 epochNumber=1
{"provingJobId":"1:BASE_PARITY:432e8f4d8b2a2404"}
13:33:05 [13:33:05.159]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.160]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3834ad0e3160f2d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3834ad0e3160f2d2"}
13:33:05 [13:33:05.161]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxyaPWd/1 with map size 1073741824
13:33:05 [13:33:05.168]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.188]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3834ad0e3160f2d2 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3834ad0e3160f2d2"}
13:33:05 [13:33:05.193]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.194]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7dfc47e1bdd88391 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7dfc47e1bdd88391"}
13:33:05 [13:33:05.194]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlJEw9D/1 with map size 1073741824
13:33:05 [13:33:05.197]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.222]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7dfc47e1bdd88391 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7dfc47e1bdd88391"}
13:33:05 [13:33:05.223]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7dfc47e1bdd88391 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7dfc47e1bdd88391"}
13:33:05 [13:33:05.233]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7dfc47e1bdd88391 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7dfc47e1bdd88391"}
13:33:05 [13:33:05.237]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1d24c28887f686f epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1d24c28887f686f"}
13:33:05 [13:33:05.238]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIJn2Hy/1 with map size 1073741824
13:33:05 [13:33:05.240]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.275]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.280]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0aab69eee72b71d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0aab69eee72b71d7"}
13:33:05 [13:33:05.283]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6tvnmH/1 with map size 1073741824
13:33:05 [13:33:05.284]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.306]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0aab69eee72b71d7
{"provingJobId":"1:BASE_PARITY:0aab69eee72b71d7"}
13:33:05 [13:33:05.306]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0aab69eee72b71d7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0aab69eee72b71d7"}
13:33:05 [13:33:05.306]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0aab69eee72b71d7 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:0aab69eee72b71d7"}
13:33:05 [13:33:05.316]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.316]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61675fc922dcf167 epochNumber=1
{"provingJobId":"1:BASE_PARITY:61675fc922dcf167"}
13:33:05 [13:33:05.317]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAboGbU/1 with map size 1073741824
13:33:05 [13:33:05.320]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.344]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:61675fc922dcf167
{"provingJobId":"1:BASE_PARITY:61675fc922dcf167"}
13:33:05 [13:33:05.345]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:61675fc922dcf167 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:61675fc922dcf167"}
13:33:05 [13:33:05.361]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c03a182b3a39d874 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c03a182b3a39d874"}
13:33:05 [13:33:05.365]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpH6dAW/1 with map size 1073741824
13:33:05 [13:33:05.372]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.394]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c03a182b3a39d874 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c03a182b3a39d874"}
13:33:05 [13:33:05.394]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c03a182b3a39d874 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c03a182b3a39d874"}
13:33:05 [13:33:05.403]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.405]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9265520734dcdd6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9265520734dcdd6"}
13:33:05 [13:33:05.406]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDH4bC6/1 with map size 1073741824
13:33:05 [13:33:05.408]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.423]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b9265520734dcdd6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b9265520734dcdd6"}
13:33:05 [13:33:05.423]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b9265520734dcdd6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b9265520734dcdd6"}
13:33:05 [13:33:05.432]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.433]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b12e095721ae3870 epochNumber=0
{"provingJobId":"1:BASE_PARITY:b12e095721ae3870"}
13:33:05 [13:33:05.434]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testMaDWop/0 with map size 1073741824
13:33:05 [13:33:05.436]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.452]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b12e095721ae3870 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b12e095721ae3870"}
13:33:05 [13:33:05.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ed8514380089c87 epochNumber=0
{"provingJobId":"1:BASE_PARITY:2ed8514380089c87"}
13:33:05 [13:33:05.453]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b12e095721ae3870 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b12e095721ae3870"}
13:33:05 [13:33:05.454]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:320f0fbe81fed08d epochNumber=0
{"provingJobId":"1:BASE_PARITY:320f0fbe81fed08d"}
13:33:05 [13:33:05.478]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.481]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.481]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1282d72e36486462 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1282d72e36486462"}
13:33:05 [13:33:05.482]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqyGEYO/2 with map size 1073741824
13:33:05 [13:33:05.485]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.503]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9e54db87f5ac35c epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9e54db87f5ac35c"}
13:33:05 [13:33:05.504]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqyGEYO/1 with map size 1073741824
13:33:05 [13:33:05.506]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.526]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:021556d42da77350 epochNumber=1
{"provingJobId":"1:BASE_PARITY:021556d42da77350"}
13:33:05 [13:33:05.527]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAyoe8w/1 with map size 1073741824
13:33:05 [13:33:05.529]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.551]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.551]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2a3ae9a0f450acc epochNumber=1
{"provingJobId":"1:BASE_PARITY:e2a3ae9a0f450acc"}
13:33:05 [13:33:05.552]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlqL3FO/1 with map size 1073741824
13:33:05 [13:33:05.566]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.586]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:966aa2074e4b6aa3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:966aa2074e4b6aa3"}
13:33:05 [13:33:05.592]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b21ddeabd3debd12 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b21ddeabd3debd12"}
13:33:05 [13:33:05.593]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlqL3FO/2 with map size 1073741824
13:33:05 [13:33:05.595]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.614]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf39c65e10e7fa20 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf39c65e10e7fa20"}
13:33:05 [13:33:05.625]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.626]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1607b8ecd9217bf4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1607b8ecd9217bf4"}
13:33:05 [13:33:05.629]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4OtLvm/1 with map size 1073741824
13:33:05 [13:33:05.631]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.649]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f32f9af6c32d289 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f32f9af6c32d289"}
13:33:05 [13:33:05.660]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c47735866b3820a8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c47735866b3820a8"}
13:33:05 [13:33:05.663]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4OtLvm/2 with map size 1073741824
13:33:05 [13:33:05.665]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.689]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fef1aa6674db7e1e epochNumber=1
{"provingJobId":"1:BASE_PARITY:fef1aa6674db7e1e"}
13:33:05 [13:33:05.699]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3daf33682cbd0145 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3daf33682cbd0145"}
13:33:05 [13:33:05.701]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testg6KePK/1 with map size 1073741824
13:33:05 [13:33:05.705]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.728]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75ef8ec1b5bf0b8e epochNumber=1
{"provingJobId":"1:BASE_PARITY:75ef8ec1b5bf0b8e"}
13:33:05 [13:33:05.739]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8410e0b5828ee628 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8410e0b5828ee628"}
13:33:05 [13:33:05.741]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testg6KePK/2 with map size 1073741824
13:33:05 [13:33:05.744]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:156547baca0ee403 epochNumber=1
{"provingJobId":"1:BASE_PARITY:156547baca0ee403"}
13:33:05 [13:33:05.782]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f707ceaddb5d2db4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f707ceaddb5d2db4"}
13:33:05 [13:33:05.787]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDvsBGm/1 with map size 1073741824
13:33:05 [13:33:05.789]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae42a9b969192a1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae42a9b969192a1f"}
13:33:05 [13:33:05.812]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f707ceaddb5d2db4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f707ceaddb5d2db4"}
13:33:05 [13:33:05.822]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.822]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3818e6542b3db87d epochNumber=1
{"provingJobId":"1:BASE_PARITY:3818e6542b3db87d"}
13:33:05 [13:33:05.823]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUMG4Gj/1 with map size 1073741824
13:33:05 [13:33:05.826]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.844]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5160c48b38681389 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5160c48b38681389"}
13:33:05 [13:33:05.851]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3818e6542b3db87d type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:3818e6542b3db87d"}
13:33:05 [13:33:05.869]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.870]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:65f4d8302bf7d20d epochNumber=1
{"provingJobId":"1:BASE_PARITY:65f4d8302bf7d20d"}
13:33:05 [13:33:05.874]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testW8v6XE/1 with map size 1073741824
13:33:05 [13:33:05.876]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.894]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a69817f282c25135 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a69817f282c25135"}
13:33:05 [13:33:05.901]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:65f4d8302bf7d20d type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:65f4d8302bf7d20d"}
13:33:05 [13:33:05.906]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.907]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8be2f9f37fe9609 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e8be2f9f37fe9609"}
13:33:05 [13:33:05.909]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqct49l/1 with map size 1073741824
13:33:05 [13:33:05.911]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.930]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e8be2f9f37fe9609
{"provingJobId":"1:BASE_PARITY:e8be2f9f37fe9609"}
13:33:05 [13:33:05.931]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e8be2f9f37fe9609 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:e8be2f9f37fe9609"}
13:33:05 [13:33:05.938]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd374c3953e8ba28 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cd374c3953e8ba28"}
13:33:05 [13:33:05.944]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e8be2f9f37fe9609 has already been completed
{"provingJobId":"1:BASE_PARITY:e8be2f9f37fe9609"}
13:33:05 [13:33:05.948]
INFO:
prover-client:proving-broker Proving Broker started
13:33:05 [13:33:05.949]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71793648f76b60b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:71793648f76b60b5"}
13:33:05 [13:33:05.950]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testabzHMm/1 with map size 1073741824
13:33:05 [13:33:05.952]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:05 [13:33:05.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36cfa53bc7fadefa epochNumber=2
{"provingJobId":"1:BASE_PARITY:36cfa53bc7fadefa"}
13:33:05 [13:33:05.976]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testabzHMm/2 with map size 1073741824
13:33:05 [13:33:05.978]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:07 [13:33:07.045]
INFO:
prover-client:proving-broker Proving Broker started
13:33:07 [13:33:07.050]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:71793648f76b60b5 settled=false
{"provingJobId":"1:BASE_PARITY:71793648f76b60b5","status":"pending"}
13:33:07 [13:33:07.051]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:36cfa53bc7fadefa settled=false
{"provingJobId":"1:BASE_PARITY:36cfa53bc7fadefa","status":"pending"}
13:33:07 [13:33:07.052]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:71793648f76b60b5 type=BASE_PARITY startedAt=1750080785993 older agent has taken job
{"provingJobId":"1:BASE_PARITY:71793648f76b60b5"}
13:33:07 [13:33:07.052]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:71793648f76b60b5 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:71793648f76b60b5"}
13:33:07 [13:33:07.059]
INFO:
prover-client:proving-broker Proving Broker started
13:33:07 [13:33:07.060]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb528b62bca6a55d epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb528b62bca6a55d"}
13:33:07 [13:33:07.060]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLERmIc/1 with map size 1073741824
13:33:07 [13:33:07.062]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:07 [13:33:07.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd9d5c784e895d9a epochNumber=2
{"provingJobId":"1:BASE_PARITY:fd9d5c784e895d9a"}
13:33:07 [13:33:07.088]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testLERmIc/2 with map size 1073741824
13:33:07 [13:33:07.091]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:08 [13:33:08.113]
INFO:
prover-client:proving-broker Proving Broker started
13:33:08 [13:33:08.116]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fb528b62bca6a55d settled=false
{"provingJobId":"1:BASE_PARITY:fb528b62bca6a55d","status":"pending"}
13:33:08 [13:33:08.120]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fd9d5c784e895d9a settled=false
{"provingJobId":"1:BASE_PARITY:fd9d5c784e895d9a","status":"pending"}
13:33:08 [13:33:08.121]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fb528b62bca6a55d type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:fb528b62bca6a55d"}
13:33:08 [13:33:08.125]
INFO:
prover-client:proving-broker Proving Broker started
13:33:08 [13:33:08.126]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f6b59d88616e2d0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:f6b59d88616e2d0f"}
13:33:08 [13:33:08.126]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testR8YrCT/1 with map size 1073741824
13:33:08 [13:33:08.128]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:08 [13:33:08.146]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52d2acaf87d8145d epochNumber=2
{"provingJobId":"1:BASE_PARITY:52d2acaf87d8145d"}
13:33:08 [13:33:08.147]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testR8YrCT/2 with map size 1073741824
13:33:08 [13:33:08.149]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:09 [13:33:09.165]
INFO:
prover-client:proving-broker Proving Broker started
13:33:09 [13:33:09.169]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f6b59d88616e2d0f settled=false
{"provingJobId":"1:BASE_PARITY:f6b59d88616e2d0f","status":"pending"}
13:33:09 [13:33:09.174]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:52d2acaf87d8145d settled=false
{"provingJobId":"1:BASE_PARITY:52d2acaf87d8145d","status":"pending"}
13:33:09 [13:33:09.177]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f6b59d88616e2d0f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f6b59d88616e2d0f"}
13:33:09 [13:33:09.177]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f6b59d88616e2d0f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f6b59d88616e2d0f"}
13:33:09 [13:33:09.191]
INFO:
prover-client:proving-broker Proving Broker started
13:33:09 [13:33:09.192]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7048eb65214333c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7048eb65214333c1"}
13:33:09 [13:33:09.193]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaCwsiM/1 with map size 1073741824
13:33:09 [13:33:09.196]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:09 [13:33:09.218]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f54650b2d2c39f5a epochNumber=2
{"provingJobId":"1:BASE_PARITY:f54650b2d2c39f5a"}
13:33:09 [13:33:09.219]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testaCwsiM/2 with map size 1073741824
13:33:09 [13:33:09.221]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:09 [13:33:09.244]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7048eb65214333c1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7048eb65214333c1"}
13:33:09 [13:33:09.252]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f54650b2d2c39f5a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f54650b2d2c39f5a"}
13:33:09 [13:33:09.271]
INFO:
prover-client:proving-broker Proving Broker started
13:33:09 [13:33:09.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fd95bdcc2201c2e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fd95bdcc2201c2e"}
13:33:09 [13:33:09.276]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjDikn4/1 with map size 1073741824
13:33:09 [13:33:09.279]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:09 [13:33:09.296]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f116f07a141fc39 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1f116f07a141fc39"}
13:33:09 [13:33:09.297]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjDikn4/2 with map size 1073741824
13:33:09 [13:33:09.301]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:09 [13:33:09.322]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4fd95bdcc2201c2e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4fd95bdcc2201c2e"}
13:33:09 [13:33:09.323]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4fd95bdcc2201c2e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4fd95bdcc2201c2e"}
13:33:09 [13:33:09.329]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1f116f07a141fc39 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1f116f07a141fc39"}
13:33:09 [13:33:09.341]
INFO:
prover-client:proving-broker Proving Broker started
13:33:09 [13:33:09.343]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1bb2a8d9b6e6b640 err=test error
{"provingJoId":"1:BASE_PARITY:1bb2a8d9b6e6b640"}
13:33:09 [13:33:09.346]
INFO:
prover-client:proving-broker Proving Broker started
13:33:09 [13:33:09.348]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:88a1ad550bdbfec4 not found
{"provingJobId":"1:BASE_PARITY:88a1ad550bdbfec4"}
13:33:09 [13:33:09.351]
INFO:
prover-client:proving-broker Proving Broker started
13:33:09 [13:33:09.352]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e1a8d7d0284c915 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e1a8d7d0284c915"}
13:33:09 [13:33:09.356]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testONN62L/1 with map size 1073741824
13:33:09 [13:33:09.363]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:09 [13:33:09.392]
INFO:
prover-client:proving-broker Proving Broker started
13:33:09 [13:33:09.393]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c31ffda7faa6d4c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c31ffda7faa6d4c5"}
13:33:09 [13:33:09.393]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGztBxo/1 with map size 1073741824
13:33:09 [13:33:09.396]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:09 [13:33:09.526]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c31ffda7faa6d4c5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c31ffda7faa6d4c5"}
13:33:09 [13:33:09.532]
INFO:
prover-client:proving-broker Proving Broker started
13:33:09 [13:33:09.533]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d2b04bdbfb0b21e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d2b04bdbfb0b21e"}
13:33:09 [13:33:09.533]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwcJfHV/1 with map size 1073741824
13:33:09 [13:33:09.536]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:09 [13:33:09.661]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4d2b04bdbfb0b21e timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4d2b04bdbfb0b21e"}
13:33:09 [13:33:09.664]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1533fd2d34fba72 epochNumber=10
{"provingJobId":"1:BASE_PARITY:c1533fd2d34fba72"}
13:33:09 [13:33:09.671]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testwcJfHV/10 with map size 1073741824
13:33:09 [13:33:09.674]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:09 [13:33:09.714]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:09 [13:33:09.714]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:33:09 [13:33:09.824]
INFO:
prover-client:proving-broker Proving Broker started
13:33:09 [13:33:09.826]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f5fad40133a5041 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f5fad40133a5041"}
13:33:09 [13:33:09.829]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlnAfDV/1 with map size 1073741824
13:33:09 [13:33:09.832]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.003]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2f5fad40133a5041 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2f5fad40133a5041"}
13:33:10 [13:33:10.057]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.058]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3570e7e8402d0007 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3570e7e8402d0007"}
13:33:10 [13:33:10.059]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test338bf8/1 with map size 1073741824
13:33:10 [13:33:10.062]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.085]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3570e7e8402d0007 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3570e7e8402d0007"}
13:33:10 [13:33:10.090]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.091]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:402e833344b1942e epochNumber=1
{"provingJobId":"1:BASE_PARITY:402e833344b1942e"}
13:33:10 [13:33:10.092]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test15tb7c/1 with map size 1073741824
13:33:10 [13:33:10.095]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.117]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:402e833344b1942e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:402e833344b1942e"}
13:33:10 [13:33:10.118]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:402e833344b1942e type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:402e833344b1942e"}
13:33:10 [13:33:10.129]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.130]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08146fe0aa04b05d epochNumber=1
{"provingJobId":"1:BASE_PARITY:08146fe0aa04b05d"}
13:33:10 [13:33:10.131]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIQ4E4h/1 with map size 1073741824
13:33:10 [13:33:10.133]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.150]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:08146fe0aa04b05d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:08146fe0aa04b05d"}
13:33:10 [13:33:10.161]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.162]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02c084572788d681 epochNumber=1
{"provingJobId":"1:BASE_PARITY:02c084572788d681"}
13:33:10 [13:33:10.164]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAcDlVW/1 with map size 1073741824
13:33:10 [13:33:10.169]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.194]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:02c084572788d681 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:02c084572788d681"}
13:33:10 [13:33:10.195]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d88893d9df5e5d36 epochNumber=3
{"provingJobId":"1:BASE_PARITY:d88893d9df5e5d36"}
13:33:10 [13:33:10.196]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testAcDlVW/3 with map size 1073741824
13:33:10 [13:33:10.197]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.238]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:10 [13:33:10.239]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:33:10 [13:33:10.241]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:02c084572788d681 err=test error
{"provingJoId":"1:BASE_PARITY:02c084572788d681"}
13:33:10 [13:33:10.246]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgok2sp/1 with map size 1073741824
13:33:10 [13:33:10.248]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.267]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testgok2sp/2 with map size 1073741824
13:33:10 [13:33:10.269]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.284]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.286]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b987ec1d01ae1361 settled=false
{"provingJobId":"1:BASE_PARITY:b987ec1d01ae1361","status":"pending"}
13:33:10 [13:33:10.287]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d95946802b7f0c3d settled=false
{"provingJobId":"1:BASE_PARITY:d95946802b7f0c3d","status":"pending"}
13:33:10 [13:33:10.295]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1kEPyT/1 with map size 1073741824
13:33:10 [13:33:10.296]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.318]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test1kEPyT/2 with map size 1073741824
13:33:10 [13:33:10.321]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.354]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.356]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8dbec70ac6bfc6f4 settled=true
{"provingJobId":"1:BASE_PARITY:8dbec70ac6bfc6f4","status":"fulfilled"}
13:33:10 [13:33:10.358]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:099500ab5575d89e settled=true
{"provingJobId":"2:BASE_PARITY:099500ab5575d89e","status":"fulfilled"}
13:33:10 [13:33:10.362]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhpFaR1/1 with map size 1073741824
13:33:10 [13:33:10.364]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.388]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testhpFaR1/2 with map size 1073741824
13:33:10 [13:33:10.390]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.407]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.408]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0b43c355c2238c82 settled=true
{"provingJobId":"1:BASE_PARITY:0b43c355c2238c82","status":"fulfilled"}
13:33:10 [13:33:10.410]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7f90c40f9c7720eb settled=false
{"provingJobId":"1:BASE_PARITY:7f90c40f9c7720eb","status":"pending"}
13:33:10 [13:33:10.414]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.415]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2bb72794f080c766 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2bb72794f080c766"}
13:33:10 [13:33:10.415]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyg3Exg/1 with map size 1073741824
13:33:10 [13:33:10.417]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.440]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.441]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e21086c4b264a3f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e21086c4b264a3f8"}
13:33:10 [13:33:10.442]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:e21086c4b264a3f8: Error: db error: Error: db error
13:33:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:e21086c4b264a3f8"}
13:33:10 [13:33:10.449]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.450]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ac895682be834c44 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ac895682be834c44"}
13:33:10 [13:33:10.451]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdN5Cur/1 with map size 1073741824
13:33:10 [13:33:10.457]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.479]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ac895682be834c44 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ac895682be834c44"}
13:33:10 [13:33:10.480]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ac895682be834c44 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ac895682be834c44"}
13:33:10 [13:33:10.496]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.497]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba113cdbe646e3b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba113cdbe646e3b0"}
13:33:10 [13:33:10.498]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbvMCva/1 with map size 1073741824
13:33:10 [13:33:10.500]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.519]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ba113cdbe646e3b0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ba113cdbe646e3b0"}
13:33:10 [13:33:10.520]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ba113cdbe646e3b0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ba113cdbe646e3b0"}
13:33:10 [13:33:10.520]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:ba113cdbe646e3b0: Error: db error
13:33:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:ba113cdbe646e3b0"}
13:33:10 [13:33:10.525]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.526]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4624aaf0572ab78b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4624aaf0572ab78b"}
13:33:10 [13:33:10.526]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfixJs8/1 with map size 1073741824
13:33:10 [13:33:10.530]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.551]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4624aaf0572ab78b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4624aaf0572ab78b"}
13:33:10 [13:33:10.552]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4624aaf0572ab78b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4624aaf0572ab78b"}
13:33:10 [13:33:10.574]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.575]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40ddc16be2861c2d epochNumber=1
{"provingJobId":"1:BASE_PARITY:40ddc16be2861c2d"}
13:33:10 [13:33:10.576]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtb9Cjx/1 with map size 1073741824
13:33:10 [13:33:10.579]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.602]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:40ddc16be2861c2d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:40ddc16be2861c2d"}
13:33:10 [13:33:10.602]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:40ddc16be2861c2d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:40ddc16be2861c2d"}
13:33:10 [13:33:10.603]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:40ddc16be2861c2d jobErr=test error: Error: db error
13:33:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:40ddc16be2861c2d"}
13:33:10 [13:33:10.607]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.608]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:405510124f2b4b51 not found
{"provingJobId":"1:BASE_PARITY:405510124f2b4b51"}
13:33:10 [13:33:10.612]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.613]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6556d78ad5a93e05 err=test error
{"provingJoId":"1:BASE_PARITY:6556d78ad5a93e05"}
13:33:10 [13:33:10.618]
INFO:
prover-client:proving-broker Proving Broker started
13:33:10 [13:33:10.629]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ad2af4871cf24d0d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ad2af4871cf24d0d"}
13:33:10 [13:33:10.630]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDByqFi/1 with map size 1073741824
13:33:10 [13:33:10.636]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.658]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ad2af4871cf24d0d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ad2af4871cf24d0d"}
13:33:10 [13:33:10.660]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ad2af4871cf24d0d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ad2af4871cf24d0d"}
13:33:10 [13:33:10.682]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:86ad62a86bc28b57 epochNumber=2
{"provingJobId":"2:BASE_PARITY:86ad62a86bc28b57"}
13:33:10 [13:33:10.683]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDByqFi/2 with map size 1073741824
13:33:10 [13:33:10.689]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.711]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:86ad62a86bc28b57 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:86ad62a86bc28b57"}
13:33:10 [13:33:10.711]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:86ad62a86bc28b57 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:86ad62a86bc28b57"}
13:33:10 [13:33:10.781]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:ce168589406a3b32 epochNumber=3
{"provingJobId":"3:BASE_PARITY:ce168589406a3b32"}
13:33:10 [13:33:10.783]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testDByqFi/3 with map size 1073741824
13:33:10 [13:33:10.784]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.822]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:10 [13:33:10.822]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:33:10 [13:33:10.865]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:40f18f102b893109 epochNumber=4
{"provingJobId":"4:BASE_PARITY:40f18f102b893109"}
13:33:10 [13:33:10.868]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testDByqFi/4 with map size 1073741824
13:33:10 [13:33:10.873]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:10 [13:33:10.900]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:10 [13:33:10.901]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
13:33:10 [13:33:10.959]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:d5039a922771fd3a epochNumber=5
{"provingJobId":"5:BASE_PARITY:d5039a922771fd3a"}
13:33:10 [13:33:10.960]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testDByqFi/5 with map size 1073741824
13:33:10 [13:33:10.964]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:33:11 [13:33:11.009]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:33:11 [13:33:11.010]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
13:33:11
PASS src/proving_broker/proving_broker.test.ts (
13.313 s)
13:33:11 ProvingBroker
13:33:11 Producer API
13:33:11
✓ refuses stale jobs (40 ms)
13:33:11
✓ enqueues jobs (4 ms)
13:33:11
✓ ignores duplicate jobs (3 ms)
13:33:11
✓ reports correct status when enqueuing repeat jobs (4 ms)
13:33:11
✓ throws an error in case of duplicate job IDs (19 ms)
13:33:11
✓ returns not-found status for non-existing jobs (10 ms)
13:33:11
✓ cancels jobs in queue (4 ms)
13:33:11
✓ cancels jobs in-progress (2 ms)
13:33:11
✓ returns job result if successful (7 ms)
13:33:11
✓ returns job error if failed (4 ms)
13:33:11
✓ correctly returns job status for concurrent writes (3 ms)
13:33:11
✓ refuses stale jobs (54 ms)
13:33:11
✓ enqueues jobs (42 ms)
13:33:11
✓ ignores duplicate jobs (35 ms)
13:33:11
✓ reports correct status when enqueuing repeat jobs (43 ms)
13:33:11
✓ throws an error in case of duplicate job IDs (33 ms)
13:33:11
✓ returns not-found status for non-existing jobs (7 ms)
13:33:11
✓ cancels jobs in queue (36 ms)
13:33:11
✓ cancels jobs in-progress (38 ms)
13:33:11
✓ returns job result if successful (49 ms)
13:33:11
✓ returns job error if failed (28 ms)
13:33:11
✓ correctly returns job status for concurrent writes (41 ms)
13:33:11 Consumer API
13:33:11
✓ returns undefined if no jobs are available (2 ms)
13:33:11
✓ returns jobs in priority order (2 ms)
13:33:11
✓ returns undefined if no jobs are available for the given allowList (1 ms)
13:33:11
✓ returns a job if it is in the allowList (4 ms)
13:33:11
✓ returns the most important job if it is in the allowList (2 ms)
13:33:11
✓ returns any job if filter is empty (2 ms)
13:33:11
✓ returns a new job when reporting job success (2 ms)
13:33:11
✓ returns a new job when reporting permanent error (2 ms)
13:33:11
✓ returns a new job when reporting retry-able error (1 ms)
13:33:11
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
13:33:11
✓ returns a new job if job is already in progress elsewhere (1055 ms)
13:33:11
✓ avoids sending the same job to a new agent after a restart (1005 ms)
13:33:11
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
13:33:11
✓ tracks job result if in progress (11 ms)
13:33:11
✓ tracks job result even if job is in queue (10 ms)
13:33:11
✓ ignores reported job error if unknown job (2 ms)
13:33:11
✓ ignores job result if unknown job (1 ms)
13:33:11
✓ returns undefined if no jobs are available (5 ms)
13:33:11
✓ returns jobs in priority order (45 ms)
13:33:11
✓ returns undefined if no jobs are available for the given allowList (24 ms)
13:33:11
✓ returns a job if it is in the allowList (74 ms)
13:33:11
✓ returns the most important job if it is in the allowList (72 ms)
13:33:11
✓ returns any job if filter is empty (81 ms)
13:33:11
✓ returns a new job when reporting job success (41 ms)
13:33:11
✓ returns a new job when reporting permanent error (43 ms)
13:33:11
✓ returns a new job when reporting retry-able error (41 ms)
13:33:11
✓ returns a new job when reporting progress if current one is cancelled (41 ms)
13:33:11
✓ returns a new job if job is already in progress elsewhere (1108 ms)
13:33:11
✓ avoids sending the same job to a new agent after a restart (1068 ms)
13:33:11
✓ avoids sending a completed job to a new agent after a restart (1065 ms)
13:33:11
✓ tracks job result if in progress (79 ms)
13:33:11
✓ tracks job result even if job is in queue (69 ms)
13:33:11
✓ ignores reported job error if unknown job (6 ms)
13:33:11
✓ ignores job result if unknown job (5 ms)
13:33:11 Timeouts
13:33:11
✓ tracks in progress jobs (2 ms)
13:33:11
✓ re-enqueues jobs that time out (103 ms)
13:33:11
✓ cancel stale jobs that time out (238 ms)
13:33:11
✓ keeps the jobs in progress while it is alive (205 ms)
13:33:11
✓ tracks in progress jobs (40 ms)
13:33:11
✓ re-enqueues jobs that time out (139 ms)
13:33:11
✓ cancel stale jobs that time out (292 ms)
13:33:11
✓ keeps the jobs in progress while it is alive (232 ms)
13:33:11 Retries
13:33:11
✓ retries jobs (2 ms)
13:33:11
✓ retries up to a maximum number of times (3 ms)
13:33:11
✓ passing retry=false does not retry (2 ms)
13:33:11
✓ does not retry if job is stale (28 ms)
13:33:11
✓ retries jobs (33 ms)
13:33:11
✓ retries up to a maximum number of times (39 ms)
13:33:11
✓ passing retry=false does not retry (30 ms)
13:33:11
✓ does not retry if job is stale (86 ms)
13:33:11 Database management
13:33:11
✓ re-enqueues proof requests on start (4 ms)
13:33:11
✓ restores proof results on start (3 ms)
13:33:11
✓ only re-enqueues unfinished jobs (2 ms)
13:33:11
✓ saves job when enqueued (5 ms)
13:33:11
✓ does not retain job if database fails to save (2 ms)
13:33:11
✓ saves job result (2 ms)
13:33:11
✓ saves result even if database fails to save (3 ms)
13:33:11
✓ saves job error (2 ms)
13:33:11
✓ saves job error even if database fails to save (2 ms)
13:33:11
✓ does not save job result if job is unknown (2 ms)
13:33:11
✓ does not save job error if job is unknown (1 ms)
13:33:11
✓ cleans up old jobs periodically (263 ms)
13:33:11
✓ re-enqueues proof requests on start (48 ms)
13:33:11
✓ restores proof results on start (67 ms)
13:33:11
✓ only re-enqueues unfinished jobs (51 ms)
13:33:11
✓ saves job when enqueued (23 ms)
13:33:11
✓ does not retain job if database fails to save (7 ms)
13:33:11
✓ saves job result (47 ms)
13:33:11
✓ saves result even if database fails to save (29 ms)
13:33:11
✓ saves job error (43 ms)
13:33:11
✓ saves job error even if database fails to save (38 ms)
13:33:11
✓ does not save job result if job is unknown (4 ms)
13:33:11
✓ does not save job error if job is unknown (5 ms)
13:33:11
✓ cleans up old jobs periodically (426 ms)
13:33:11
13:33:11
Test Suites: 1 passed, 1 total
13:33:11
Tests: 96 passed, 96 total
13:33:11
Snapshots: 0 total
13:33:11
Time: 13.446 s
13:33:11
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
13:33:11
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?