Parent Log:
http://ci.aztec-labs.com/a3904b8892ad0957
Command: f9856415d54196c8 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/3084a7eb89277f14c021ec43f4bddf01c77cc5cc
Env: REF_NAME=gh-readonly-queue/next/pr-15077-433ba6218aed45169029991a493e4232b30b8ec6 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 17 19:12:35 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15077_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
19:12:39 [19:12:39.511]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86ec4268ad561568 epochNumber=42
{"provingJobId":"1:BASE_PARITY:86ec4268ad561568"}
19:12:39 [19:12:39.524]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:120b4fc16e6f7d60 epochNumber=1
{"provingJobId":"1:BASE_PARITY:120b4fc16e6f7d60"}
19:12:39 [19:12:39.563]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.564]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89fc8a25e4c7c360 epochNumber=1
{"provingJobId":"1:BASE_PARITY:89fc8a25e4c7c360"}
19:12:39 [19:12:39.565]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7517abd21cad689c epochNumber=1
{"provingJobId":"1:BASE_PARITY:7517abd21cad689c"}
19:12:39 [19:12:39.568]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.569]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68545fb490b5e743 epochNumber=1
{"provingJobId":"1:BASE_PARITY:68545fb490b5e743"}
19:12:39 [19:12:39.571]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:68545fb490b5e743 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:68545fb490b5e743"}
19:12:39 [19:12:39.574]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.575]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a14f25fd5dc4a91 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a14f25fd5dc4a91"}
19:12:39 [19:12:39.576]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1a14f25fd5dc4a91 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1a14f25fd5dc4a91"}
19:12:39 [19:12:39.577]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1a14f25fd5dc4a91 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1a14f25fd5dc4a91"}
19:12:39 [19:12:39.578]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1a14f25fd5dc4a91 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1a14f25fd5dc4a91"}
19:12:39 [19:12:39.580]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.581]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56cf63c436dee211 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56cf63c436dee211"}
19:12:39 [19:12:39.615]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.617]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.617]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d01568258643ca0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d01568258643ca0"}
19:12:39 [19:12:39.618]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6d01568258643ca0
{"provingJobId":"1:BASE_PARITY:6d01568258643ca0"}
19:12:39 [19:12:39.619]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6d01568258643ca0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6d01568258643ca0"}
19:12:39 [19:12:39.619]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6d01568258643ca0 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:6d01568258643ca0"}
19:12:39 [19:12:39.621]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.621]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbb7f0ae61e13645 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dbb7f0ae61e13645"}
19:12:39 [19:12:39.622]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:dbb7f0ae61e13645
{"provingJobId":"1:BASE_PARITY:dbb7f0ae61e13645"}
19:12:39 [19:12:39.622]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dbb7f0ae61e13645 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:dbb7f0ae61e13645"}
19:12:39 [19:12:39.623]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.624]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64ba939f8a270583 epochNumber=1
{"provingJobId":"1:BASE_PARITY:64ba939f8a270583"}
19:12:39 [19:12:39.625]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:64ba939f8a270583 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:64ba939f8a270583"}
19:12:39 [19:12:39.625]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:64ba939f8a270583 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:64ba939f8a270583"}
19:12:39 [19:12:39.626]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.627]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:313b082600e2458f epochNumber=1
{"provingJobId":"1:BASE_PARITY:313b082600e2458f"}
19:12:39 [19:12:39.627]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:313b082600e2458f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:313b082600e2458f"}
19:12:39 [19:12:39.628]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:313b082600e2458f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:313b082600e2458f"}
19:12:39 [19:12:39.629]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.629]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3af2beafbfe40867 epochNumber=0
{"provingJobId":"1:BASE_PARITY:3af2beafbfe40867"}
19:12:39 [19:12:39.629]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3af2beafbfe40867 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3af2beafbfe40867"}
19:12:39 [19:12:39.629]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:010674240d6059a8 epochNumber=0
{"provingJobId":"1:BASE_PARITY:010674240d6059a8"}
19:12:39 [19:12:39.630]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3af2beafbfe40867 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3af2beafbfe40867"}
19:12:39 [19:12:39.630]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e8ad01c077a5eab epochNumber=0
{"provingJobId":"1:BASE_PARITY:6e8ad01c077a5eab"}
19:12:39 [19:12:39.631]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.632]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.632]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd2accc82557117d epochNumber=2
{"provingJobId":"1:BASE_PARITY:bd2accc82557117d"}
19:12:39 [19:12:39.632]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a5d1568213c8144f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a5d1568213c8144f"}
19:12:39 [19:12:39.633]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.633]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91ebda3b9a2a6243 epochNumber=1
{"provingJobId":"1:BASE_PARITY:91ebda3b9a2a6243"}
19:12:39 [19:12:39.634]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.634]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79aa44f66224bff6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:79aa44f66224bff6"}
19:12:39 [19:12:39.635]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87f8e66fdcb14d64 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87f8e66fdcb14d64"}
19:12:39 [19:12:39.635]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dea4b3c7f23f40c9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dea4b3c7f23f40c9"}
19:12:39 [19:12:39.635]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:618e8fbb7af3f6cf epochNumber=1
{"provingJobId":"1:BASE_PARITY:618e8fbb7af3f6cf"}
19:12:39 [19:12:39.636]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.636]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c5332783cb803c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c5332783cb803c4"}
19:12:39 [19:12:39.636]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:717d357708da9559 epochNumber=1
{"provingJobId":"1:BASE_PARITY:717d357708da9559"}
19:12:39 [19:12:39.637]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b3b004f7548ac43 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6b3b004f7548ac43"}
19:12:39 [19:12:39.637]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a65944df34e3c8d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a65944df34e3c8d2"}
19:12:39 [19:12:39.638]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.638]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58fe38f0e61fb40a epochNumber=1
{"provingJobId":"1:BASE_PARITY:58fe38f0e61fb40a"}
19:12:39 [19:12:39.638]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f27daf0c5ed49760 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f27daf0c5ed49760"}
19:12:39 [19:12:39.638]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56553a5dbae6a66e epochNumber=2
{"provingJobId":"1:BASE_PARITY:56553a5dbae6a66e"}
19:12:39 [19:12:39.639]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5737f1f8a037ec96 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5737f1f8a037ec96"}
19:12:39 [19:12:39.639]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.640]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6db8df9cc1ea1c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b6db8df9cc1ea1c3"}
19:12:39 [19:12:39.640]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46ad5facd55f0377 epochNumber=1
{"provingJobId":"1:BASE_PARITY:46ad5facd55f0377"}
19:12:39 [19:12:39.640]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b6db8df9cc1ea1c3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b6db8df9cc1ea1c3"}
19:12:39 [19:12:39.641]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.641]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:299047e8dde7a0e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:299047e8dde7a0e3"}
19:12:39 [19:12:39.642]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26ee4ec8b476287c epochNumber=1
{"provingJobId":"1:BASE_PARITY:26ee4ec8b476287c"}
19:12:39 [19:12:39.642]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:299047e8dde7a0e3 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:299047e8dde7a0e3"}
19:12:39 [19:12:39.643]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.643]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d10c6c8000a2be3e epochNumber=1
{"provingJobId":"1:BASE_PARITY:d10c6c8000a2be3e"}
19:12:39 [19:12:39.643]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8edf7f1d4aaf17f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8edf7f1d4aaf17f3"}
19:12:39 [19:12:39.644]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d10c6c8000a2be3e type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:d10c6c8000a2be3e"}
19:12:39 [19:12:39.644]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.645]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:006abc7915145967 epochNumber=1
{"provingJobId":"1:BASE_PARITY:006abc7915145967"}
19:12:39 [19:12:39.645]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:006abc7915145967
{"provingJobId":"1:BASE_PARITY:006abc7915145967"}
19:12:39 [19:12:39.645]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:006abc7915145967 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:006abc7915145967"}
19:12:39 [19:12:39.646]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb2cae78c2e35dec epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb2cae78c2e35dec"}
19:12:39 [19:12:39.646]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:006abc7915145967 has already been completed
{"provingJobId":"1:BASE_PARITY:006abc7915145967"}
19:12:39 [19:12:39.647]
INFO:
prover-client:proving-broker Proving Broker started
19:12:39 [19:12:39.648]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b047d04cdd6fa75 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b047d04cdd6fa75"}
19:12:39 [19:12:39.648]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d821356f6e1e318 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5d821356f6e1e318"}
19:12:40 [19:12:40.699]
INFO:
prover-client:proving-broker Proving Broker started
19:12:40 [19:12:40.700]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9b047d04cdd6fa75 settled=false
{"provingJobId":"1:BASE_PARITY:9b047d04cdd6fa75","status":"pending"}
19:12:40 [19:12:40.700]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5d821356f6e1e318 settled=false
{"provingJobId":"1:BASE_PARITY:5d821356f6e1e318","status":"pending"}
19:12:40 [19:12:40.701]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9b047d04cdd6fa75 type=BASE_PARITY startedAt=1750187559648 older agent has taken job
{"provingJobId":"1:BASE_PARITY:9b047d04cdd6fa75"}
19:12:40 [19:12:40.701]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9b047d04cdd6fa75 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:9b047d04cdd6fa75"}
19:12:40 [19:12:40.702]
INFO:
prover-client:proving-broker Proving Broker started
19:12:40 [19:12:40.703]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7aa9eaebab9da0d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7aa9eaebab9da0d3"}
19:12:40 [19:12:40.703]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c61a269c09cc85b epochNumber=2
{"provingJobId":"1:BASE_PARITY:2c61a269c09cc85b"}
19:12:41 [19:12:41.704]
INFO:
prover-client:proving-broker Proving Broker started
19:12:41 [19:12:41.705]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7aa9eaebab9da0d3 settled=false
{"provingJobId":"1:BASE_PARITY:7aa9eaebab9da0d3","status":"pending"}
19:12:41 [19:12:41.705]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2c61a269c09cc85b settled=false
{"provingJobId":"1:BASE_PARITY:2c61a269c09cc85b","status":"pending"}
19:12:41 [19:12:41.706]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7aa9eaebab9da0d3 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:7aa9eaebab9da0d3"}
19:12:41 [19:12:41.707]
INFO:
prover-client:proving-broker Proving Broker started
19:12:41 [19:12:41.708]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:63411d482f717a4a epochNumber=1
{"provingJobId":"1:BASE_PARITY:63411d482f717a4a"}
19:12:41 [19:12:41.709]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f6160fb9b897c4ad epochNumber=2
{"provingJobId":"1:BASE_PARITY:f6160fb9b897c4ad"}
19:12:42 [19:12:42.710]
INFO:
prover-client:proving-broker Proving Broker started
19:12:42 [19:12:42.711]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:63411d482f717a4a settled=false
{"provingJobId":"1:BASE_PARITY:63411d482f717a4a","status":"pending"}
19:12:42 [19:12:42.711]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f6160fb9b897c4ad settled=false
{"provingJobId":"1:BASE_PARITY:f6160fb9b897c4ad","status":"pending"}
19:12:42 [19:12:42.712]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:63411d482f717a4a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:63411d482f717a4a"}
19:12:42 [19:12:42.712]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:63411d482f717a4a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:63411d482f717a4a"}
19:12:42 [19:12:42.714]
INFO:
prover-client:proving-broker Proving Broker started
19:12:42 [19:12:42.714]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d329c95ab336d288 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d329c95ab336d288"}
19:12:42 [19:12:42.714]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e5b0b6b1b7eab0e epochNumber=2
{"provingJobId":"1:BASE_PARITY:1e5b0b6b1b7eab0e"}
19:12:42 [19:12:42.715]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d329c95ab336d288 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d329c95ab336d288"}
19:12:42 [19:12:42.716]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1e5b0b6b1b7eab0e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1e5b0b6b1b7eab0e"}
19:12:42 [19:12:42.717]
INFO:
prover-client:proving-broker Proving Broker started
19:12:42 [19:12:42.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6412f91f5b9a63bb epochNumber=1
{"provingJobId":"1:BASE_PARITY:6412f91f5b9a63bb"}
19:12:42 [19:12:42.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d705ccffada8328e epochNumber=2
{"provingJobId":"1:BASE_PARITY:d705ccffada8328e"}
19:12:42 [19:12:42.718]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6412f91f5b9a63bb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6412f91f5b9a63bb"}
19:12:42 [19:12:42.718]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6412f91f5b9a63bb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6412f91f5b9a63bb"}
19:12:42 [19:12:42.718]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d705ccffada8328e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d705ccffada8328e"}
19:12:42 [19:12:42.719]
INFO:
prover-client:proving-broker Proving Broker started
19:12:42 [19:12:42.720]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:74e2920b68a62817 err=test error
{"provingJoId":"1:BASE_PARITY:74e2920b68a62817"}
19:12:42 [19:12:42.721]
INFO:
prover-client:proving-broker Proving Broker started
19:12:42 [19:12:42.721]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:90313069b48ccf65 not found
{"provingJobId":"1:BASE_PARITY:90313069b48ccf65"}
19:12:42 [19:12:42.723]
INFO:
prover-client:proving-broker Proving Broker started
19:12:42 [19:12:42.724]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfc749c6fa04232c epochNumber=1
{"provingJobId":"1:BASE_PARITY:dfc749c6fa04232c"}
19:12:42 [19:12:42.726]
INFO:
prover-client:proving-broker Proving Broker started
19:12:42 [19:12:42.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e82eb1739fb2f3c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e82eb1739fb2f3c4"}
19:12:42 [19:12:42.829]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e82eb1739fb2f3c4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e82eb1739fb2f3c4"}
19:12:42 [19:12:42.837]
INFO:
prover-client:proving-broker Proving Broker started
19:12:42 [19:12:42.838]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71b9d1642581d597 epochNumber=1
{"provingJobId":"1:BASE_PARITY:71b9d1642581d597"}
19:12:42 [19:12:42.967]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:71b9d1642581d597 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:71b9d1642581d597"}
19:12:42 [19:12:42.974]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0e7a8b55af13a4a epochNumber=10
{"provingJobId":"1:BASE_PARITY:f0e7a8b55af13a4a"}
19:12:42 [19:12:42.993]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:43 [19:12:43.101]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.102]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4286b61d9e86ebf epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4286b61d9e86ebf"}
19:12:43 [19:12:43.255]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c4286b61d9e86ebf timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c4286b61d9e86ebf"}
19:12:43 [19:12:43.306]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.307]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf33f3c5310861be epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf33f3c5310861be"}
19:12:43 [19:12:43.308]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cf33f3c5310861be type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:cf33f3c5310861be"}
19:12:43 [19:12:43.309]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.310]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4dfa1883e1c7ba33 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4dfa1883e1c7ba33"}
19:12:43 [19:12:43.310]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4dfa1883e1c7ba33 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:4dfa1883e1c7ba33"}
19:12:43 [19:12:43.311]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4dfa1883e1c7ba33 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:4dfa1883e1c7ba33"}
19:12:43 [19:12:43.312]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.312]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d946ffc38c9d233b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d946ffc38c9d233b"}
19:12:43 [19:12:43.312]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d946ffc38c9d233b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d946ffc38c9d233b"}
19:12:43 [19:12:43.313]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.314]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7eedd6a63f47ddba epochNumber=1
{"provingJobId":"1:BASE_PARITY:7eedd6a63f47ddba"}
19:12:43 [19:12:43.314]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:7eedd6a63f47ddba type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:7eedd6a63f47ddba"}
19:12:43 [19:12:43.315]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88cb7e5d32845ae5 epochNumber=3
{"provingJobId":"1:BASE_PARITY:88cb7e5d32845ae5"}
19:12:43 [19:12:43.339]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:43 [19:12:43.339]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7eedd6a63f47ddba err=test error
{"provingJoId":"1:BASE_PARITY:7eedd6a63f47ddba"}
19:12:43 [19:12:43.341]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.341]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fc9d968bcbd8d88d settled=false
{"provingJobId":"1:BASE_PARITY:fc9d968bcbd8d88d","status":"pending"}
19:12:43 [19:12:43.341]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2b4d2acebc63943f settled=false
{"provingJobId":"1:BASE_PARITY:2b4d2acebc63943f","status":"pending"}
19:12:43 [19:12:43.343]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.343]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3245efafad81e2dc settled=true
{"provingJobId":"1:BASE_PARITY:3245efafad81e2dc","status":"fulfilled"}
19:12:43 [19:12:43.343]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:fb0952b41a4300ac settled=true
{"provingJobId":"2:BASE_PARITY:fb0952b41a4300ac","status":"fulfilled"}
19:12:43 [19:12:43.345]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.345]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e44e77413aa44c0c settled=true
{"provingJobId":"1:BASE_PARITY:e44e77413aa44c0c","status":"fulfilled"}
19:12:43 [19:12:43.345]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b567e751984cef66 settled=false
{"provingJobId":"1:BASE_PARITY:b567e751984cef66","status":"pending"}
19:12:43 [19:12:43.346]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.347]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b1d096c684e7a1ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:b1d096c684e7a1ea"}
19:12:43 [19:12:43.349]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.349]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2d565dd506d6534 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2d565dd506d6534"}
19:12:43 [19:12:43.350]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:f2d565dd506d6534: Error: db error: Error: db error
19:12:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:f2d565dd506d6534"}
19:12:43 [19:12:43.352]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.352]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:090471cfb662fced epochNumber=1
{"provingJobId":"1:BASE_PARITY:090471cfb662fced"}
19:12:43 [19:12:43.352]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:090471cfb662fced type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:090471cfb662fced"}
19:12:43 [19:12:43.353]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:090471cfb662fced type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:090471cfb662fced"}
19:12:43 [19:12:43.354]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.354]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01e4d085dc249e82 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01e4d085dc249e82"}
19:12:43 [19:12:43.355]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:01e4d085dc249e82 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:01e4d085dc249e82"}
19:12:43 [19:12:43.355]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:01e4d085dc249e82 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:01e4d085dc249e82"}
19:12:43 [19:12:43.356]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:01e4d085dc249e82: Error: db error
19:12:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:01e4d085dc249e82"}
19:12:43 [19:12:43.357]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.358]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:540f146ef729cb2a epochNumber=1
{"provingJobId":"1:BASE_PARITY:540f146ef729cb2a"}
19:12:43 [19:12:43.358]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:540f146ef729cb2a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:540f146ef729cb2a"}
19:12:43 [19:12:43.358]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:540f146ef729cb2a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:540f146ef729cb2a"}
19:12:43 [19:12:43.360]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.360]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b701edd3c9aa412 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b701edd3c9aa412"}
19:12:43 [19:12:43.361]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b701edd3c9aa412 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7b701edd3c9aa412"}
19:12:43 [19:12:43.361]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7b701edd3c9aa412 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7b701edd3c9aa412"}
19:12:43 [19:12:43.361]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:7b701edd3c9aa412 jobErr=test error: Error: db error
19:12:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:7b701edd3c9aa412"}
19:12:43 [19:12:43.363]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.363]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6015d4e75194f1c5 not found
{"provingJobId":"1:BASE_PARITY:6015d4e75194f1c5"}
19:12:43 [19:12:43.365]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.365]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:965aed30d7f412a0 err=test error
{"provingJoId":"1:BASE_PARITY:965aed30d7f412a0"}
19:12:43 [19:12:43.366]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.376]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66986d556dd9297c epochNumber=1
{"provingJobId":"1:BASE_PARITY:66986d556dd9297c"}
19:12:43 [19:12:43.377]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:66986d556dd9297c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:66986d556dd9297c"}
19:12:43 [19:12:43.377]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:66986d556dd9297c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:66986d556dd9297c"}
19:12:43 [19:12:43.388]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:45460f88fa36bdbf epochNumber=2
{"provingJobId":"2:BASE_PARITY:45460f88fa36bdbf"}
19:12:43 [19:12:43.389]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:45460f88fa36bdbf type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:45460f88fa36bdbf"}
19:12:43 [19:12:43.390]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:45460f88fa36bdbf type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:45460f88fa36bdbf"}
19:12:43 [19:12:43.449]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:63d8d98b0d714bc5 epochNumber=3
{"provingJobId":"3:BASE_PARITY:63d8d98b0d714bc5"}
19:12:43 [19:12:43.468]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:43 [19:12:43.513]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:eecdee72403abb9d epochNumber=4
{"provingJobId":"4:BASE_PARITY:eecdee72403abb9d"}
19:12:43 [19:12:43.519]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:43 [19:12:43.578]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:a3ba95acd597dbf4 epochNumber=5
{"provingJobId":"5:BASE_PARITY:a3ba95acd597dbf4"}
19:12:43 [19:12:43.597]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:43 [19:12:43.633]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.634]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9c0752b92ff5c33 epochNumber=42
{"provingJobId":"1:BASE_PARITY:b9c0752b92ff5c33"}
19:12:43 [19:12:43.635]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testDYvJAL/42 with map size 1073741824
19:12:43 [19:12:43.641]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.675]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:948dc2dec72b8668 epochNumber=1
{"provingJobId":"1:BASE_PARITY:948dc2dec72b8668"}
19:12:43 [19:12:43.683]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.684]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0f45f44c5569b1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:f0f45f44c5569b1f"}
19:12:43 [19:12:43.686]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFdoNDJ/1 with map size 1073741824
19:12:43 [19:12:43.690]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.719]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34793521bcc1463a epochNumber=1
{"provingJobId":"1:BASE_PARITY:34793521bcc1463a"}
19:12:43 [19:12:43.729]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.730]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e211674d824cb9a epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e211674d824cb9a"}
19:12:43 [19:12:43.732]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKuyrXY/1 with map size 1073741824
19:12:43 [19:12:43.734]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.751]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8e211674d824cb9a epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8e211674d824cb9a"}
19:12:43 [19:12:43.754]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.755]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d16154bddc761380 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d16154bddc761380"}
19:12:43 [19:12:43.755]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4xfov8/1 with map size 1073741824
19:12:43 [19:12:43.757]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.780]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d16154bddc761380 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d16154bddc761380"}
19:12:43 [19:12:43.781]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d16154bddc761380 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d16154bddc761380"}
19:12:43 [19:12:43.789]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d16154bddc761380 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d16154bddc761380"}
19:12:43 [19:12:43.793]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.794]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9cfba6772572e3b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9cfba6772572e3b"}
19:12:43 [19:12:43.795]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTWGqsq/1 with map size 1073741824
19:12:43 [19:12:43.798]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.822]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.824]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.825]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:270585e984bce177 epochNumber=1
{"provingJobId":"1:BASE_PARITY:270585e984bce177"}
19:12:43 [19:12:43.825]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWyQg1H/1 with map size 1073741824
19:12:43 [19:12:43.827]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.845]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:270585e984bce177
{"provingJobId":"1:BASE_PARITY:270585e984bce177"}
19:12:43 [19:12:43.845]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:270585e984bce177 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:270585e984bce177"}
19:12:43 [19:12:43.846]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:270585e984bce177 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:270585e984bce177"}
19:12:43 [19:12:43.854]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.855]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ddd0a2888d281e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ddd0a2888d281e1"}
19:12:43 [19:12:43.855]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDLFOBU/1 with map size 1073741824
19:12:43 [19:12:43.857]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.875]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4ddd0a2888d281e1
{"provingJobId":"1:BASE_PARITY:4ddd0a2888d281e1"}
19:12:43 [19:12:43.876]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4ddd0a2888d281e1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4ddd0a2888d281e1"}
19:12:43 [19:12:43.885]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.886]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c01b36e5512ff50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c01b36e5512ff50"}
19:12:43 [19:12:43.886]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYV4bq3/1 with map size 1073741824
19:12:43 [19:12:43.888]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.906]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8c01b36e5512ff50 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8c01b36e5512ff50"}
19:12:43 [19:12:43.907]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8c01b36e5512ff50 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8c01b36e5512ff50"}
19:12:43 [19:12:43.914]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.914]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47c092d8f3d5f01b epochNumber=1
{"provingJobId":"1:BASE_PARITY:47c092d8f3d5f01b"}
19:12:43 [19:12:43.915]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBKjrvr/1 with map size 1073741824
19:12:43 [19:12:43.917]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.936]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:47c092d8f3d5f01b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:47c092d8f3d5f01b"}
19:12:43 [19:12:43.937]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:47c092d8f3d5f01b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:47c092d8f3d5f01b"}
19:12:43 [19:12:43.947]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.947]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16cca45029d2a8d2 epochNumber=0
{"provingJobId":"1:BASE_PARITY:16cca45029d2a8d2"}
19:12:43 [19:12:43.949]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test595oIr/0 with map size 1073741824
19:12:43 [19:12:43.951]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:43 [19:12:43.969]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:16cca45029d2a8d2 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:16cca45029d2a8d2"}
19:12:43 [19:12:43.970]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b726b961d1f03c9c epochNumber=0
{"provingJobId":"1:BASE_PARITY:b726b961d1f03c9c"}
19:12:43 [19:12:43.970]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:16cca45029d2a8d2 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:16cca45029d2a8d2"}
19:12:43 [19:12:43.970]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:264f84339f98bf91 epochNumber=0
{"provingJobId":"1:BASE_PARITY:264f84339f98bf91"}
19:12:43 [19:12:43.987]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.990]
INFO:
prover-client:proving-broker Proving Broker started
19:12:43 [19:12:43.991]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:810f88ea6ea96de4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:810f88ea6ea96de4"}
19:12:43 [19:12:43.991]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQQzAI0/2 with map size 1073741824
19:12:43 [19:12:43.993]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6d00cf7acbbdecf epochNumber=1
{"provingJobId":"1:BASE_PARITY:b6d00cf7acbbdecf"}
19:12:44 [19:12:44.013]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQQzAI0/1 with map size 1073741824
19:12:44 [19:12:44.014]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.035]
INFO:
prover-client:proving-broker Proving Broker started
19:12:44 [19:12:44.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22e47ebbf9ed8436 epochNumber=1
{"provingJobId":"1:BASE_PARITY:22e47ebbf9ed8436"}
19:12:44 [19:12:44.036]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhDp0pU/1 with map size 1073741824
19:12:44 [19:12:44.038]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.060]
INFO:
prover-client:proving-broker Proving Broker started
19:12:44 [19:12:44.061]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:caee18a70822196c epochNumber=1
{"provingJobId":"1:BASE_PARITY:caee18a70822196c"}
19:12:44 [19:12:44.062]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7xrXr1/1 with map size 1073741824
19:12:44 [19:12:44.064]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1df15c377a631f7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c1df15c377a631f7"}
19:12:44 [19:12:44.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:767b82d42e76d5f8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:767b82d42e76d5f8"}
19:12:44 [19:12:44.078]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7xrXr1/2 with map size 1073741824
19:12:44 [19:12:44.080]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.104]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d7e04163b4c1612 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5d7e04163b4c1612"}
19:12:44 [19:12:44.116]
INFO:
prover-client:proving-broker Proving Broker started
19:12:44 [19:12:44.117]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ed5d83b0f586af8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ed5d83b0f586af8"}
19:12:44 [19:12:44.117]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOdpyiR/1 with map size 1073741824
19:12:44 [19:12:44.121]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.141]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5ae88fa024b525b epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5ae88fa024b525b"}
19:12:44 [19:12:44.148]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20856211aecda0da epochNumber=2
{"provingJobId":"1:BASE_PARITY:20856211aecda0da"}
19:12:44 [19:12:44.149]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOdpyiR/2 with map size 1073741824
19:12:44 [19:12:44.151]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fce08746f33b82b epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fce08746f33b82b"}
19:12:44 [19:12:44.177]
INFO:
prover-client:proving-broker Proving Broker started
19:12:44 [19:12:44.178]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64270342d3360d01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:64270342d3360d01"}
19:12:44 [19:12:44.180]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrixLER/1 with map size 1073741824
19:12:44 [19:12:44.182]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.201]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1492c749be01e65d epochNumber=1
{"provingJobId":"1:BASE_PARITY:1492c749be01e65d"}
19:12:44 [19:12:44.206]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be1b88d557f161c2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:be1b88d557f161c2"}
19:12:44 [19:12:44.207]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrixLER/2 with map size 1073741824
19:12:44 [19:12:44.209]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.227]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7036dd1305dec48b epochNumber=1
{"provingJobId":"1:BASE_PARITY:7036dd1305dec48b"}
19:12:44 [19:12:44.236]
INFO:
prover-client:proving-broker Proving Broker started
19:12:44 [19:12:44.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfef374b10c892f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dfef374b10c892f3"}
19:12:44 [19:12:44.237]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzCDfZx/1 with map size 1073741824
19:12:44 [19:12:44.240]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54b669416eb9a075 epochNumber=1
{"provingJobId":"1:BASE_PARITY:54b669416eb9a075"}
19:12:44 [19:12:44.275]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dfef374b10c892f3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dfef374b10c892f3"}
19:12:44 [19:12:44.286]
INFO:
prover-client:proving-broker Proving Broker started
19:12:44 [19:12:44.287]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7fd869a9f0fa4245 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7fd869a9f0fa4245"}
19:12:44 [19:12:44.288]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-teste0Zyk2/1 with map size 1073741824
19:12:44 [19:12:44.291]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.307]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3120ed4d69ce060b epochNumber=1
{"provingJobId":"1:BASE_PARITY:3120ed4d69ce060b"}
19:12:44 [19:12:44.314]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7fd869a9f0fa4245 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:7fd869a9f0fa4245"}
19:12:44 [19:12:44.326]
INFO:
prover-client:proving-broker Proving Broker started
19:12:44 [19:12:44.327]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf392b7992e3c013 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf392b7992e3c013"}
19:12:44 [19:12:44.327]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYywIJd/1 with map size 1073741824
19:12:44 [19:12:44.329]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.349]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff8728173eead4d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff8728173eead4d1"}
19:12:44 [19:12:44.356]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cf392b7992e3c013 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:cf392b7992e3c013"}
19:12:44 [19:12:44.361]
INFO:
prover-client:proving-broker Proving Broker started
19:12:44 [19:12:44.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce83794f1ff4bd82 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ce83794f1ff4bd82"}
19:12:44 [19:12:44.363]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testORTiwK/1 with map size 1073741824
19:12:44 [19:12:44.366]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.384]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ce83794f1ff4bd82
{"provingJobId":"1:BASE_PARITY:ce83794f1ff4bd82"}
19:12:44 [19:12:44.385]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ce83794f1ff4bd82 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ce83794f1ff4bd82"}
19:12:44 [19:12:44.393]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ea39f297579fd0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ea39f297579fd0f"}
19:12:44 [19:12:44.400]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ce83794f1ff4bd82 has already been completed
{"provingJobId":"1:BASE_PARITY:ce83794f1ff4bd82"}
19:12:44 [19:12:44.405]
INFO:
prover-client:proving-broker Proving Broker started
19:12:44 [19:12:44.407]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36106b8bb5044377 epochNumber=1
{"provingJobId":"1:BASE_PARITY:36106b8bb5044377"}
19:12:44 [19:12:44.407]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBhFCxw/1 with map size 1073741824
19:12:44 [19:12:44.409]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:44 [19:12:44.428]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b36dc7b85d119d8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1b36dc7b85d119d8"}
19:12:44 [19:12:44.430]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testBhFCxw/2 with map size 1073741824
19:12:44 [19:12:44.432]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:45 [19:12:45.501]
INFO:
prover-client:proving-broker Proving Broker started
19:12:45 [19:12:45.508]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:36106b8bb5044377 settled=false
{"provingJobId":"1:BASE_PARITY:36106b8bb5044377","status":"pending"}
19:12:45 [19:12:45.510]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1b36dc7b85d119d8 settled=false
{"provingJobId":"1:BASE_PARITY:1b36dc7b85d119d8","status":"pending"}
19:12:45 [19:12:45.512]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:36106b8bb5044377 type=BASE_PARITY startedAt=1750187564449 older agent has taken job
{"provingJobId":"1:BASE_PARITY:36106b8bb5044377"}
19:12:45 [19:12:45.512]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:36106b8bb5044377 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:36106b8bb5044377"}
19:12:45 [19:12:45.517]
INFO:
prover-client:proving-broker Proving Broker started
19:12:45 [19:12:45.518]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd9ca86078551328 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd9ca86078551328"}
19:12:45 [19:12:45.518]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4JoVVI/1 with map size 1073741824
19:12:45 [19:12:45.520]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:45 [19:12:45.539]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1346f882d82c91a9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1346f882d82c91a9"}
19:12:45 [19:12:45.540]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4JoVVI/2 with map size 1073741824
19:12:45 [19:12:45.543]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:46 [19:12:46.564]
INFO:
prover-client:proving-broker Proving Broker started
19:12:46 [19:12:46.566]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fd9ca86078551328 settled=false
{"provingJobId":"1:BASE_PARITY:fd9ca86078551328","status":"pending"}
19:12:46 [19:12:46.567]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1346f882d82c91a9 settled=false
{"provingJobId":"1:BASE_PARITY:1346f882d82c91a9","status":"pending"}
19:12:46 [19:12:46.568]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd9ca86078551328 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:fd9ca86078551328"}
19:12:46 [19:12:46.573]
INFO:
prover-client:proving-broker Proving Broker started
19:12:46 [19:12:46.574]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0761ea3e04a5ab51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0761ea3e04a5ab51"}
19:12:46 [19:12:46.575]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6Au6lZ/1 with map size 1073741824
19:12:46 [19:12:46.577]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:46 [19:12:46.597]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c95edb243b4b456d epochNumber=2
{"provingJobId":"1:BASE_PARITY:c95edb243b4b456d"}
19:12:46 [19:12:46.598]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6Au6lZ/2 with map size 1073741824
19:12:46 [19:12:46.600]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:47 [19:12:47.617]
INFO:
prover-client:proving-broker Proving Broker started
19:12:47 [19:12:47.620]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0761ea3e04a5ab51 settled=false
{"provingJobId":"1:BASE_PARITY:0761ea3e04a5ab51","status":"pending"}
19:12:47 [19:12:47.623]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c95edb243b4b456d settled=false
{"provingJobId":"1:BASE_PARITY:c95edb243b4b456d","status":"pending"}
19:12:47 [19:12:47.624]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0761ea3e04a5ab51 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0761ea3e04a5ab51"}
19:12:47 [19:12:47.625]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0761ea3e04a5ab51 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0761ea3e04a5ab51"}
19:12:47 [19:12:47.636]
INFO:
prover-client:proving-broker Proving Broker started
19:12:47 [19:12:47.636]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88688752d9d147c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:88688752d9d147c6"}
19:12:47 [19:12:47.637]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpR39Ee/1 with map size 1073741824
19:12:47 [19:12:47.640]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:47 [19:12:47.659]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:883473e54fe9fceb epochNumber=2
{"provingJobId":"1:BASE_PARITY:883473e54fe9fceb"}
19:12:47 [19:12:47.660]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpR39Ee/2 with map size 1073741824
19:12:47 [19:12:47.667]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:47 [19:12:47.687]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:88688752d9d147c6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:88688752d9d147c6"}
19:12:47 [19:12:47.694]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:883473e54fe9fceb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:883473e54fe9fceb"}
19:12:47 [19:12:47.708]
INFO:
prover-client:proving-broker Proving Broker started
19:12:47 [19:12:47.710]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd128bccd0e19912 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd128bccd0e19912"}
19:12:47 [19:12:47.710]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3HojkR/1 with map size 1073741824
19:12:47 [19:12:47.712]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:47 [19:12:47.732]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ac47da4d3f338f7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2ac47da4d3f338f7"}
19:12:47 [19:12:47.733]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test3HojkR/2 with map size 1073741824
19:12:47 [19:12:47.735]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:47 [19:12:47.758]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd128bccd0e19912 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fd128bccd0e19912"}
19:12:47 [19:12:47.759]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fd128bccd0e19912 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fd128bccd0e19912"}
19:12:47 [19:12:47.766]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2ac47da4d3f338f7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2ac47da4d3f338f7"}
19:12:47 [19:12:47.781]
INFO:
prover-client:proving-broker Proving Broker started
19:12:47 [19:12:47.782]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:77b860b83110f5c2 err=test error
{"provingJoId":"1:BASE_PARITY:77b860b83110f5c2"}
19:12:47 [19:12:47.789]
INFO:
prover-client:proving-broker Proving Broker started
19:12:47 [19:12:47.790]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d0b6a3f72f5901ed not found
{"provingJobId":"1:BASE_PARITY:d0b6a3f72f5901ed"}
19:12:47 [19:12:47.795]
INFO:
prover-client:proving-broker Proving Broker started
19:12:47 [19:12:47.795]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8346593169094eb7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8346593169094eb7"}
19:12:47 [19:12:47.796]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxNIzCC/1 with map size 1073741824
19:12:47 [19:12:47.801]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:47 [19:12:47.822]
INFO:
prover-client:proving-broker Proving Broker started
19:12:47 [19:12:47.823]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:223745f594d5572c epochNumber=1
{"provingJobId":"1:BASE_PARITY:223745f594d5572c"}
19:12:47 [19:12:47.824]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWznJEA/1 with map size 1073741824
19:12:47 [19:12:47.826]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:47 [19:12:47.949]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:223745f594d5572c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:223745f594d5572c"}
19:12:47 [19:12:47.957]
INFO:
prover-client:proving-broker Proving Broker started
19:12:47 [19:12:47.958]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c0e5c3fa829c3dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:7c0e5c3fa829c3dc"}
19:12:47 [19:12:47.959]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrRpWL1/1 with map size 1073741824
19:12:47 [19:12:47.961]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.082]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7c0e5c3fa829c3dc timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:7c0e5c3fa829c3dc"}
19:12:48 [19:12:48.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:956570ff888f61e5 epochNumber=10
{"provingJobId":"1:BASE_PARITY:956570ff888f61e5"}
19:12:48 [19:12:48.087]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testrRpWL1/10 with map size 1073741824
19:12:48 [19:12:48.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.108]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:48 [19:12:48.109]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:12:48 [19:12:48.236]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.237]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ab85764f702c1c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ab85764f702c1c2"}
19:12:48 [19:12:48.237]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testB5GPS3/1 with map size 1073741824
19:12:48 [19:12:48.239]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.414]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9ab85764f702c1c2 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9ab85764f702c1c2"}
19:12:48 [19:12:48.462]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.463]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:153f33aef30d9a78 epochNumber=1
{"provingJobId":"1:BASE_PARITY:153f33aef30d9a78"}
19:12:48 [19:12:48.464]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQOVt9S/1 with map size 1073741824
19:12:48 [19:12:48.465]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.482]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:153f33aef30d9a78 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:153f33aef30d9a78"}
19:12:48 [19:12:48.486]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.487]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3f1f28f17f616ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3f1f28f17f616ed"}
19:12:48 [19:12:48.488]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfxyVMX/1 with map size 1073741824
19:12:48 [19:12:48.490]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.507]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a3f1f28f17f616ed type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a3f1f28f17f616ed"}
19:12:48 [19:12:48.508]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a3f1f28f17f616ed type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:a3f1f28f17f616ed"}
19:12:48 [19:12:48.519]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f171922d1d8446e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f171922d1d8446e0"}
19:12:48 [19:12:48.521]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testW8Va2U/1 with map size 1073741824
19:12:48 [19:12:48.525]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.543]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f171922d1d8446e0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f171922d1d8446e0"}
19:12:48 [19:12:48.560]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.562]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:411f13caf8800a01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:411f13caf8800a01"}
19:12:48 [19:12:48.562]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZaVeDJ/1 with map size 1073741824
19:12:48 [19:12:48.565]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.583]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:411f13caf8800a01 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:411f13caf8800a01"}
19:12:48 [19:12:48.584]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:244d6d7322b76e21 epochNumber=3
{"provingJobId":"1:BASE_PARITY:244d6d7322b76e21"}
19:12:48 [19:12:48.585]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testZaVeDJ/3 with map size 1073741824
19:12:48 [19:12:48.587]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.611]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:48 [19:12:48.611]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:12:48 [19:12:48.629]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:411f13caf8800a01 err=test error
{"provingJoId":"1:BASE_PARITY:411f13caf8800a01"}
19:12:48 [19:12:48.633]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPV6gN7/1 with map size 1073741824
19:12:48 [19:12:48.635]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.658]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testPV6gN7/2 with map size 1073741824
19:12:48 [19:12:48.660]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.680]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.682]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b506f5a3c08d3791 settled=false
{"provingJobId":"1:BASE_PARITY:b506f5a3c08d3791","status":"pending"}
19:12:48 [19:12:48.684]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ef18fb5b6a62a750 settled=false
{"provingJobId":"1:BASE_PARITY:ef18fb5b6a62a750","status":"pending"}
19:12:48 [19:12:48.689]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGir4bC/1 with map size 1073741824
19:12:48 [19:12:48.692]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.710]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGir4bC/2 with map size 1073741824
19:12:48 [19:12:48.714]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.748]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.760]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7d963b31a1aa0697 settled=true
{"provingJobId":"1:BASE_PARITY:7d963b31a1aa0697","status":"fulfilled"}
19:12:48 [19:12:48.773]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:61627200778d8518 settled=true
{"provingJobId":"2:BASE_PARITY:61627200778d8518","status":"fulfilled"}
19:12:48 [19:12:48.789]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpx1kH6/1 with map size 1073741824
19:12:48 [19:12:48.793]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.820]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpx1kH6/2 with map size 1073741824
19:12:48 [19:12:48.824]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.852]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.855]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9c480d1b6abaebe2 settled=true
{"provingJobId":"1:BASE_PARITY:9c480d1b6abaebe2","status":"fulfilled"}
19:12:48 [19:12:48.858]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:806c02ccc7762bb5 settled=false
{"provingJobId":"1:BASE_PARITY:806c02ccc7762bb5","status":"pending"}
19:12:48 [19:12:48.875]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.877]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2530266b9c7b4849 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2530266b9c7b4849"}
19:12:48 [19:12:48.878]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6OSMwg/1 with map size 1073741824
19:12:48 [19:12:48.882]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.908]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.909]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b44436e23381e06b epochNumber=1
{"provingJobId":"1:BASE_PARITY:b44436e23381e06b"}
19:12:48 [19:12:48.910]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:b44436e23381e06b: Error: db error: Error: db error
19:12:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:b44436e23381e06b"}
19:12:48 [19:12:48.913]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.913]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3233266e61d00d11 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3233266e61d00d11"}
19:12:48 [19:12:48.914]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testubnVjk/1 with map size 1073741824
19:12:48 [19:12:48.916]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.934]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3233266e61d00d11 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3233266e61d00d11"}
19:12:48 [19:12:48.935]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3233266e61d00d11 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3233266e61d00d11"}
19:12:48 [19:12:48.947]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.948]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94d63c3819c12490 epochNumber=1
{"provingJobId":"1:BASE_PARITY:94d63c3819c12490"}
19:12:48 [19:12:48.949]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuIBvKS/1 with map size 1073741824
19:12:48 [19:12:48.952]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:48 [19:12:48.979]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:94d63c3819c12490 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:94d63c3819c12490"}
19:12:48 [19:12:48.980]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:94d63c3819c12490 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:94d63c3819c12490"}
19:12:48 [19:12:48.982]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:94d63c3819c12490: Error: db error
19:12:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:94d63c3819c12490"}
19:12:48 [19:12:48.990]
INFO:
prover-client:proving-broker Proving Broker started
19:12:48 [19:12:48.991]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d27d23abd960492c epochNumber=1
{"provingJobId":"1:BASE_PARITY:d27d23abd960492c"}
19:12:48 [19:12:48.991]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxSBqT8/1 with map size 1073741824
19:12:48 [19:12:48.993]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:49 [19:12:49.012]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d27d23abd960492c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d27d23abd960492c"}
19:12:49 [19:12:49.013]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d27d23abd960492c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d27d23abd960492c"}
19:12:49 [19:12:49.024]
INFO:
prover-client:proving-broker Proving Broker started
19:12:49 [19:12:49.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b7036f6bb11dbe9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b7036f6bb11dbe9"}
19:12:49 [19:12:49.026]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJ1y6ix/1 with map size 1073741824
19:12:49 [19:12:49.029]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:49 [19:12:49.052]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8b7036f6bb11dbe9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8b7036f6bb11dbe9"}
19:12:49 [19:12:49.052]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8b7036f6bb11dbe9 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8b7036f6bb11dbe9"}
19:12:49 [19:12:49.053]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:8b7036f6bb11dbe9 jobErr=test error: Error: db error
19:12:49 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:8b7036f6bb11dbe9"}
19:12:49 [19:12:49.060]
INFO:
prover-client:proving-broker Proving Broker started
19:12:49 [19:12:49.061]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:00d2fde9f9d78974 not found
{"provingJobId":"1:BASE_PARITY:00d2fde9f9d78974"}
19:12:49 [19:12:49.067]
INFO:
prover-client:proving-broker Proving Broker started
19:12:49 [19:12:49.068]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c8ed19202842407f err=test error
{"provingJoId":"1:BASE_PARITY:c8ed19202842407f"}
19:12:49 [19:12:49.078]
INFO:
prover-client:proving-broker Proving Broker started
19:12:49 [19:12:49.090]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1604e000ee6cc5c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1604e000ee6cc5c7"}
19:12:49 [19:12:49.091]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcnQ7vq/1 with map size 1073741824
19:12:49 [19:12:49.095]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:49 [19:12:49.114]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1604e000ee6cc5c7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1604e000ee6cc5c7"}
19:12:49 [19:12:49.114]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1604e000ee6cc5c7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1604e000ee6cc5c7"}
19:12:49 [19:12:49.132]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:43e01156a2b8f0fd epochNumber=2
{"provingJobId":"2:BASE_PARITY:43e01156a2b8f0fd"}
19:12:49 [19:12:49.133]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcnQ7vq/2 with map size 1073741824
19:12:49 [19:12:49.136]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:49 [19:12:49.158]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:43e01156a2b8f0fd type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:43e01156a2b8f0fd"}
19:12:49 [19:12:49.159]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:43e01156a2b8f0fd type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:43e01156a2b8f0fd"}
19:12:49 [19:12:49.230]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:36530596edb81a88 epochNumber=3
{"provingJobId":"3:BASE_PARITY:36530596edb81a88"}
19:12:49 [19:12:49.232]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testcnQ7vq/3 with map size 1073741824
19:12:49 [19:12:49.236]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:49 [19:12:49.259]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:49 [19:12:49.262]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:12:49 [19:12:49.313]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:b8683494dd129393 epochNumber=4
{"provingJobId":"4:BASE_PARITY:b8683494dd129393"}
19:12:49 [19:12:49.314]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testcnQ7vq/4 with map size 1073741824
19:12:49 [19:12:49.317]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:49 [19:12:49.355]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:49 [19:12:49.356]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
19:12:49 [19:12:49.399]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:562826630dd130aa epochNumber=5
{"provingJobId":"5:BASE_PARITY:562826630dd130aa"}
19:12:49 [19:12:49.400]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testcnQ7vq/5 with map size 1073741824
19:12:49 [19:12:49.402]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:12:49 [19:12:49.441]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:12:49 [19:12:49.442]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
19:12:49
PASS src/proving_broker/proving_broker.test.ts (
13.363 s)
19:12:49 ProvingBroker
19:12:49 Producer API
19:12:49
✓ refuses stale jobs (53 ms)
19:12:49
✓ enqueues jobs (5 ms)
19:12:49
✓ ignores duplicate jobs (5 ms)
19:12:49
✓ reports correct status when enqueuing repeat jobs (6 ms)
19:12:49
✓ throws an error in case of duplicate job IDs (35 ms)
19:12:49
✓ returns not-found status for non-existing jobs (1 ms)
19:12:49
✓ cancels jobs in queue (4 ms)
19:12:49
✓ cancels jobs in-progress (3 ms)
19:12:49
✓ returns job result if successful (2 ms)
19:12:49
✓ returns job error if failed (2 ms)
19:12:49
✓ correctly returns job status for concurrent writes (2 ms)
19:12:49
✓ refuses stale jobs (48 ms)
19:12:49
✓ enqueues jobs (47 ms)
19:12:49
✓ ignores duplicate jobs (26 ms)
19:12:49
✓ reports correct status when enqueuing repeat jobs (38 ms)
19:12:49
✓ throws an error in case of duplicate job IDs (29 ms)
19:12:49
✓ returns not-found status for non-existing jobs (3 ms)
19:12:49
✓ cancels jobs in queue (30 ms)
19:12:49
✓ cancels jobs in-progress (30 ms)
19:12:49
✓ returns job result if successful (29 ms)
19:12:49
✓ returns job error if failed (32 ms)
19:12:49
✓ correctly returns job status for concurrent writes (39 ms)
19:12:49 Consumer API
19:12:49
✓ returns undefined if no jobs are available (1 ms)
19:12:49
✓ returns jobs in priority order (2 ms)
19:12:49
✓ returns undefined if no jobs are available for the given allowList (1 ms)
19:12:49
✓ returns a job if it is in the allowList (1 ms)
19:12:49
✓ returns the most important job if it is in the allowList (1 ms)
19:12:49
✓ returns any job if filter is empty (2 ms)
19:12:49
✓ returns a new job when reporting job success (2 ms)
19:12:49
✓ returns a new job when reporting permanent error (1 ms)
19:12:49
✓ returns a new job when reporting retry-able error (2 ms)
19:12:49
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
19:12:49
✓ returns a new job if job is already in progress elsewhere (1055 ms)
19:12:49
✓ avoids sending the same job to a new agent after a restart (1005 ms)
19:12:49
✓ avoids sending a completed job to a new agent after a restart (1006 ms)
19:12:49
✓ tracks job result if in progress (3 ms)
19:12:49
✓ tracks job result even if job is in queue (3 ms)
19:12:49
✓ ignores reported job error if unknown job (1 ms)
19:12:49
✓ ignores job result if unknown job (2 ms)
19:12:49
✓ returns undefined if no jobs are available (3 ms)
19:12:49
✓ returns jobs in priority order (45 ms)
19:12:49
✓ returns undefined if no jobs are available for the given allowList (25 ms)
19:12:49
✓ returns a job if it is in the allowList (55 ms)
19:12:49
✓ returns the most important job if it is in the allowList (61 ms)
19:12:49
✓ returns any job if filter is empty (59 ms)
19:12:49
✓ returns a new job when reporting job success (48 ms)
19:12:49
✓ returns a new job when reporting permanent error (40 ms)
19:12:49
✓ returns a new job when reporting retry-able error (34 ms)
19:12:49
✓ returns a new job when reporting progress if current one is cancelled (44 ms)
19:12:49
✓ returns a new job if job is already in progress elsewhere (1111 ms)
19:12:49
✓ avoids sending the same job to a new agent after a restart (1056 ms)
19:12:49
✓ avoids sending a completed job to a new agent after a restart (1062 ms)
19:12:49
✓ tracks job result if in progress (69 ms)
19:12:49
✓ tracks job result even if job is in queue (76 ms)
19:12:49
✓ ignores reported job error if unknown job (3 ms)
19:12:49
✓ ignores job result if unknown job (8 ms)
19:12:49 Timeouts
19:12:49
✓ tracks in progress jobs (4 ms)
19:12:49
✓ re-enqueues jobs that time out (110 ms)
19:12:49
✓ cancel stale jobs that time out (264 ms)
19:12:49
✓ keeps the jobs in progress while it is alive (204 ms)
19:12:49
✓ tracks in progress jobs (27 ms)
19:12:49
✓ re-enqueues jobs that time out (135 ms)
19:12:49
✓ cancel stale jobs that time out (278 ms)
19:12:49
✓ keeps the jobs in progress while it is alive (225 ms)
19:12:49 Retries
19:12:49
✓ retries jobs (3 ms)
19:12:49
✓ retries up to a maximum number of times (2 ms)
19:12:49
✓ passing retry=false does not retry (2 ms)
19:12:49
✓ does not retry if job is stale (27 ms)
19:12:49
✓ retries jobs (24 ms)
19:12:49
✓ retries up to a maximum number of times (32 ms)
19:12:49
✓ passing retry=false does not retry (40 ms)
19:12:49
✓ does not retry if job is stale (74 ms)
19:12:49 Database management
19:12:49
✓ re-enqueues proof requests on start (2 ms)
19:12:49
✓ restores proof results on start (1 ms)
19:12:49
✓ only re-enqueues unfinished jobs (2 ms)
19:12:49
✓ saves job when enqueued (2 ms)
19:12:49
✓ does not retain job if database fails to save (2 ms)
19:12:49
✓ saves job result (2 ms)
19:12:49
✓ saves result even if database fails to save (2 ms)
19:12:49
✓ saves job error (2 ms)
19:12:49
✓ saves job error even if database fails to save (3 ms)
19:12:49
✓ does not save job result if job is unknown (2 ms)
19:12:49
✓ does not save job error if job is unknown (2 ms)
19:12:49
✓ cleans up old jobs periodically (264 ms)
19:12:49
✓ re-enqueues proof requests on start (56 ms)
19:12:49
✓ restores proof results on start (98 ms)
19:12:49
✓ only re-enqueues unfinished jobs (82 ms)
19:12:49
✓ saves job when enqueued (37 ms)
19:12:49
✓ does not retain job if database fails to save (4 ms)
19:12:49
✓ saves job result (31 ms)
19:12:49
✓ saves result even if database fails to save (42 ms)
19:12:49
✓ saves job error (36 ms)
19:12:49
✓ saves job error even if database fails to save (33 ms)
19:12:49
✓ does not save job result if job is unknown (8 ms)
19:12:49
✓ does not save job error if job is unknown (6 ms)
19:12:49
✓ cleans up old jobs periodically (404 ms)
19:12:49
19:12:49
Test Suites: 1 passed, 1 total
19:12:49
Tests: 96 passed, 96 total
19:12:49
Snapshots: 0 total
19:12:49
Time: 13.463 s
19:12:49
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
19:12:49
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?