Parent Log:
http://ci.aztec-labs.com/481126c5e1da4234
Command: 9d60524367dfa79a yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/056e5230c330d6ad19900135ecf2a4787181496a
Env: REF_NAME=gh-readonly-queue/next/pr-12986-a9f91dd1e5946aecfeb192ce869eb3e8eca2742e CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 17 13:13:51 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-12986_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
13:13:55 [13:13:55.972]
INFO:
prover-client:proving-broker Proving Broker started
13:13:55 [13:13:55.979]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7c6c59b98f11df6 epochNumber=42
{"provingJobId":"1:BASE_PARITY:b7c6c59b98f11df6"}
13:13:55 [13:13:55.984]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:245696bf01ce3955 epochNumber=1
{"provingJobId":"1:BASE_PARITY:245696bf01ce3955"}
13:13:56 [13:13:56.028]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be09400850ac9f58 epochNumber=1
{"provingJobId":"1:BASE_PARITY:be09400850ac9f58"}
13:13:56 [13:13:56.030]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b069906919e35f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b069906919e35f2"}
13:13:56 [13:13:56.032]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22f246f4555a1e30 epochNumber=1
{"provingJobId":"1:BASE_PARITY:22f246f4555a1e30"}
13:13:56 [13:13:56.035]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:22f246f4555a1e30 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:22f246f4555a1e30"}
13:13:56 [13:13:56.037]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.038]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97239a5ba87d111b epochNumber=1
{"provingJobId":"1:BASE_PARITY:97239a5ba87d111b"}
13:13:56 [13:13:56.040]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:97239a5ba87d111b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:97239a5ba87d111b"}
13:13:56 [13:13:56.040]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:97239a5ba87d111b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:97239a5ba87d111b"}
13:13:56 [13:13:56.041]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:97239a5ba87d111b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:97239a5ba87d111b"}
13:13:56 [13:13:56.043]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.044]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fedf53b011e8c12a epochNumber=1
{"provingJobId":"1:BASE_PARITY:fedf53b011e8c12a"}
13:13:56 [13:13:56.075]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.077]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.078]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:30afc233f51c075b epochNumber=1
{"provingJobId":"1:BASE_PARITY:30afc233f51c075b"}
13:13:56 [13:13:56.079]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:30afc233f51c075b
{"provingJobId":"1:BASE_PARITY:30afc233f51c075b"}
13:13:56 [13:13:56.080]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:30afc233f51c075b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:30afc233f51c075b"}
13:13:56 [13:13:56.080]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:30afc233f51c075b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:30afc233f51c075b"}
13:13:56 [13:13:56.081]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:048922889b4e4259 epochNumber=1
{"provingJobId":"1:BASE_PARITY:048922889b4e4259"}
13:13:56 [13:13:56.083]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:048922889b4e4259
{"provingJobId":"1:BASE_PARITY:048922889b4e4259"}
13:13:56 [13:13:56.083]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:048922889b4e4259 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:048922889b4e4259"}
13:13:56 [13:13:56.085]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42c2226daeae600e epochNumber=1
{"provingJobId":"1:BASE_PARITY:42c2226daeae600e"}
13:13:56 [13:13:56.086]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:42c2226daeae600e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:42c2226daeae600e"}
13:13:56 [13:13:56.087]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:42c2226daeae600e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:42c2226daeae600e"}
13:13:56 [13:13:56.088]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.089]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ff687984963b508 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ff687984963b508"}
13:13:56 [13:13:56.089]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8ff687984963b508 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8ff687984963b508"}
13:13:56 [13:13:56.090]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8ff687984963b508 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8ff687984963b508"}
13:13:56 [13:13:56.091]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.092]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b48e41bcdf32208 epochNumber=0
{"provingJobId":"1:BASE_PARITY:6b48e41bcdf32208"}
13:13:56 [13:13:56.092]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6b48e41bcdf32208 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6b48e41bcdf32208"}
13:13:56 [13:13:56.093]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5419cb350a8e00ff epochNumber=0
{"provingJobId":"1:BASE_PARITY:5419cb350a8e00ff"}
13:13:56 [13:13:56.093]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6b48e41bcdf32208 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6b48e41bcdf32208"}
13:13:56 [13:13:56.093]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2a3eb7d9482dfd3 epochNumber=0
{"provingJobId":"1:BASE_PARITY:e2a3eb7d9482dfd3"}
13:13:56 [13:13:56.115]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.118]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be0c9d31f6fed9cf epochNumber=2
{"provingJobId":"1:BASE_PARITY:be0c9d31f6fed9cf"}
13:13:56 [13:13:56.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5c84cb58ab34412 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c5c84cb58ab34412"}
13:13:56 [13:13:56.121]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5467238678597b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c5467238678597b8"}
13:13:56 [13:13:56.123]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa9490614ab8e0df epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa9490614ab8e0df"}
13:13:56 [13:13:56.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11d85a9f3b3175d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:11d85a9f3b3175d3"}
13:13:56 [13:13:56.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a02843113aab81b epochNumber=2
{"provingJobId":"1:BASE_PARITY:2a02843113aab81b"}
13:13:56 [13:13:56.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:baa9e4c704705b28 epochNumber=1
{"provingJobId":"1:BASE_PARITY:baa9e4c704705b28"}
13:13:56 [13:13:56.125]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.126]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff369d77e04d2d1d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff369d77e04d2d1d"}
13:13:56 [13:13:56.126]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fc1cf8b415d492f epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fc1cf8b415d492f"}
13:13:56 [13:13:56.126]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15f601929313cacf epochNumber=2
{"provingJobId":"1:BASE_PARITY:15f601929313cacf"}
13:13:56 [13:13:56.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ece141ea2112c2b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ece141ea2112c2b6"}
13:13:56 [13:13:56.128]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.128]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2de57a906d49614 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2de57a906d49614"}
13:13:56 [13:13:56.129]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:237e892752c6b9d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:237e892752c6b9d5"}
13:13:56 [13:13:56.129]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff21f689601ba6ce epochNumber=2
{"provingJobId":"1:BASE_PARITY:ff21f689601ba6ce"}
13:13:56 [13:13:56.129]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a823e98a62c8e286 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a823e98a62c8e286"}
13:13:56 [13:13:56.131]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.131]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:49d6ce4b426a7607 epochNumber=1
{"provingJobId":"1:BASE_PARITY:49d6ce4b426a7607"}
13:13:56 [13:13:56.131]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ad6a115f17a11f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3ad6a115f17a11f0"}
13:13:56 [13:13:56.132]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:49d6ce4b426a7607 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:49d6ce4b426a7607"}
13:13:56 [13:13:56.133]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:389dc6dc74ee7cf0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:389dc6dc74ee7cf0"}
13:13:56 [13:13:56.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9cfa33a137f79f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9cfa33a137f79f2"}
13:13:56 [13:13:56.134]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:389dc6dc74ee7cf0 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:389dc6dc74ee7cf0"}
13:13:56 [13:13:56.134]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.135]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:360b17936462e7d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:360b17936462e7d4"}
13:13:56 [13:13:56.135]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0bf6a0f0099cd026 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0bf6a0f0099cd026"}
13:13:56 [13:13:56.135]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:360b17936462e7d4 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:360b17936462e7d4"}
13:13:56 [13:13:56.136]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.137]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eeead566f7cbf327 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eeead566f7cbf327"}
13:13:56 [13:13:56.137]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:eeead566f7cbf327
{"provingJobId":"1:BASE_PARITY:eeead566f7cbf327"}
13:13:56 [13:13:56.137]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:eeead566f7cbf327 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:eeead566f7cbf327"}
13:13:56 [13:13:56.137]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c2aa3dffc599df8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c2aa3dffc599df8"}
13:13:56 [13:13:56.138]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eeead566f7cbf327 has already been completed
{"provingJobId":"1:BASE_PARITY:eeead566f7cbf327"}
13:13:56 [13:13:56.138]
INFO:
prover-client:proving-broker Proving Broker started
13:13:56 [13:13:56.139]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb603335d22b7767 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb603335d22b7767"}
13:13:56 [13:13:56.139]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f01c5b95923dc39b epochNumber=2
{"provingJobId":"1:BASE_PARITY:f01c5b95923dc39b"}
13:13:57 [13:13:57.192]
INFO:
prover-client:proving-broker Proving Broker started
13:13:57 [13:13:57.194]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:eb603335d22b7767 settled=false
{"provingJobId":"1:BASE_PARITY:eb603335d22b7767","status":"pending"}
13:13:57 [13:13:57.195]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f01c5b95923dc39b settled=false
{"provingJobId":"1:BASE_PARITY:f01c5b95923dc39b","status":"pending"}
13:13:57 [13:13:57.197]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb603335d22b7767 type=BASE_PARITY startedAt=1750166036139 older agent has taken job
{"provingJobId":"1:BASE_PARITY:eb603335d22b7767"}
13:13:57 [13:13:57.197]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb603335d22b7767 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:eb603335d22b7767"}
13:13:57 [13:13:57.199]
INFO:
prover-client:proving-broker Proving Broker started
13:13:57 [13:13:57.201]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd1493e2653c5358 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd1493e2653c5358"}
13:13:57 [13:13:57.201]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d050052d1a2805f3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d050052d1a2805f3"}
13:13:58 [13:13:58.203]
INFO:
prover-client:proving-broker Proving Broker started
13:13:58 [13:13:58.203]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bd1493e2653c5358 settled=false
{"provingJobId":"1:BASE_PARITY:bd1493e2653c5358","status":"pending"}
13:13:58 [13:13:58.204]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d050052d1a2805f3 settled=false
{"provingJobId":"1:BASE_PARITY:d050052d1a2805f3","status":"pending"}
13:13:58 [13:13:58.204]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bd1493e2653c5358 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:bd1493e2653c5358"}
13:13:58 [13:13:58.205]
INFO:
prover-client:proving-broker Proving Broker started
13:13:58 [13:13:58.206]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01032c2537097ba8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01032c2537097ba8"}
13:13:58 [13:13:58.206]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16563a2a7a22f6d8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:16563a2a7a22f6d8"}
13:13:59 [13:13:59.207]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.207]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:01032c2537097ba8 settled=false
{"provingJobId":"1:BASE_PARITY:01032c2537097ba8","status":"pending"}
13:13:59 [13:13:59.208]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:16563a2a7a22f6d8 settled=false
{"provingJobId":"1:BASE_PARITY:16563a2a7a22f6d8","status":"pending"}
13:13:59 [13:13:59.208]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:01032c2537097ba8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:01032c2537097ba8"}
13:13:59 [13:13:59.209]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:01032c2537097ba8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:01032c2537097ba8"}
13:13:59 [13:13:59.211]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.212]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d324c517a16c039b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d324c517a16c039b"}
13:13:59 [13:13:59.212]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b2520cce7c5f873 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9b2520cce7c5f873"}
13:13:59 [13:13:59.213]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d324c517a16c039b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d324c517a16c039b"}
13:13:59 [13:13:59.214]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9b2520cce7c5f873 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9b2520cce7c5f873"}
13:13:59 [13:13:59.215]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.215]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:727e4ba609f4b59e epochNumber=1
{"provingJobId":"1:BASE_PARITY:727e4ba609f4b59e"}
13:13:59 [13:13:59.216]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:909c62bd3c1d7b7e epochNumber=2
{"provingJobId":"1:BASE_PARITY:909c62bd3c1d7b7e"}
13:13:59 [13:13:59.216]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:727e4ba609f4b59e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:727e4ba609f4b59e"}
13:13:59 [13:13:59.216]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:727e4ba609f4b59e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:727e4ba609f4b59e"}
13:13:59 [13:13:59.217]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:909c62bd3c1d7b7e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:909c62bd3c1d7b7e"}
13:13:59 [13:13:59.218]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.218]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6ab40544123715bb err=test error
{"provingJoId":"1:BASE_PARITY:6ab40544123715bb"}
13:13:59 [13:13:59.219]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.220]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:162d0074f6d2bd89 not found
{"provingJobId":"1:BASE_PARITY:162d0074f6d2bd89"}
13:13:59 [13:13:59.221]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e6f961d755fbd74 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e6f961d755fbd74"}
13:13:59 [13:13:59.223]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.224]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d7ffdaf387e9625 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d7ffdaf387e9625"}
13:13:59 [13:13:59.325]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0d7ffdaf387e9625 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:0d7ffdaf387e9625"}
13:13:59 [13:13:59.334]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c7f7fdc2f5d390dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:c7f7fdc2f5d390dd"}
13:13:59 [13:13:59.436]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c7f7fdc2f5d390dd timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c7f7fdc2f5d390dd"}
13:13:59 [13:13:59.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ea74ba8af51af03 epochNumber=10
{"provingJobId":"1:BASE_PARITY:3ea74ba8af51af03"}
13:13:59 [13:13:59.463]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:13:59 [13:13:59.566]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.567]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b1d4f17b9e407f1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b1d4f17b9e407f1"}
13:13:59 [13:13:59.722]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4b1d4f17b9e407f1 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4b1d4f17b9e407f1"}
13:13:59 [13:13:59.773]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.774]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d546d103199f8398 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d546d103199f8398"}
13:13:59 [13:13:59.775]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d546d103199f8398 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d546d103199f8398"}
13:13:59 [13:13:59.777]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a0ef470098ce819b epochNumber=1
{"provingJobId":"1:BASE_PARITY:a0ef470098ce819b"}
13:13:59 [13:13:59.778]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a0ef470098ce819b type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a0ef470098ce819b"}
13:13:59 [13:13:59.779]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a0ef470098ce819b type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:a0ef470098ce819b"}
13:13:59 [13:13:59.781]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.781]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f0105037ec1396e epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f0105037ec1396e"}
13:13:59 [13:13:59.782]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0f0105037ec1396e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0f0105037ec1396e"}
13:13:59 [13:13:59.783]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97f0a8205b96ab1a epochNumber=1
{"provingJobId":"1:BASE_PARITY:97f0a8205b96ab1a"}
13:13:59 [13:13:59.784]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:97f0a8205b96ab1a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:97f0a8205b96ab1a"}
13:13:59 [13:13:59.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:822d623bf9b272e6 epochNumber=3
{"provingJobId":"1:BASE_PARITY:822d623bf9b272e6"}
13:13:59 [13:13:59.809]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:13:59 [13:13:59.811]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:97f0a8205b96ab1a err=test error
{"provingJoId":"1:BASE_PARITY:97f0a8205b96ab1a"}
13:13:59 [13:13:59.814]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.814]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c325e3675f624537 settled=false
{"provingJobId":"1:BASE_PARITY:c325e3675f624537","status":"pending"}
13:13:59 [13:13:59.814]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f19813dbed6c2897 settled=false
{"provingJobId":"1:BASE_PARITY:f19813dbed6c2897","status":"pending"}
13:13:59 [13:13:59.817]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.817]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f25394167db918b7 settled=true
{"provingJobId":"1:BASE_PARITY:f25394167db918b7","status":"fulfilled"}
13:13:59 [13:13:59.818]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:034e3d275beedd7b settled=true
{"provingJobId":"2:BASE_PARITY:034e3d275beedd7b","status":"fulfilled"}
13:13:59 [13:13:59.819]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.819]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f29c8c426290c912 settled=true
{"provingJobId":"1:BASE_PARITY:f29c8c426290c912","status":"fulfilled"}
13:13:59 [13:13:59.819]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8336ea467910d083 settled=false
{"provingJobId":"1:BASE_PARITY:8336ea467910d083","status":"pending"}
13:13:59 [13:13:59.821]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.824]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf5d34848fd9cf36 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf5d34848fd9cf36"}
13:13:59 [13:13:59.828]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.829]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4990d158f6379e29 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4990d158f6379e29"}
13:13:59 [13:13:59.830]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:4990d158f6379e29: Error: db error: Error: db error
13:13:59 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:4990d158f6379e29"}
13:13:59 [13:13:59.832]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.832]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a7ce8558317abba1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a7ce8558317abba1"}
13:13:59 [13:13:59.833]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a7ce8558317abba1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a7ce8558317abba1"}
13:13:59 [13:13:59.833]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a7ce8558317abba1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a7ce8558317abba1"}
13:13:59 [13:13:59.835]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.835]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:166a3f5da330fdc3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:166a3f5da330fdc3"}
13:13:59 [13:13:59.836]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:166a3f5da330fdc3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:166a3f5da330fdc3"}
13:13:59 [13:13:59.836]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:166a3f5da330fdc3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:166a3f5da330fdc3"}
13:13:59 [13:13:59.837]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:166a3f5da330fdc3: Error: db error
13:13:59 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:166a3f5da330fdc3"}
13:13:59 [13:13:59.838]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.839]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d26e1f949a7da57 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2d26e1f949a7da57"}
13:13:59 [13:13:59.840]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2d26e1f949a7da57 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2d26e1f949a7da57"}
13:13:59 [13:13:59.840]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2d26e1f949a7da57 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2d26e1f949a7da57"}
13:13:59 [13:13:59.842]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d8df34e79db7e74 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d8df34e79db7e74"}
13:13:59 [13:13:59.847]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0d8df34e79db7e74 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0d8df34e79db7e74"}
13:13:59 [13:13:59.847]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0d8df34e79db7e74 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0d8df34e79db7e74"}
13:13:59 [13:13:59.848]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:0d8df34e79db7e74 jobErr=test error: Error: db error
13:13:59 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:0d8df34e79db7e74"}
13:13:59 [13:13:59.851]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.852]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:efe568fe72e6e9f4 not found
{"provingJobId":"1:BASE_PARITY:efe568fe72e6e9f4"}
13:13:59 [13:13:59.854]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.856]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:68d7c5092e944ed1 err=test error
{"provingJoId":"1:BASE_PARITY:68d7c5092e944ed1"}
13:13:59 [13:13:59.858]
INFO:
prover-client:proving-broker Proving Broker started
13:13:59 [13:13:59.869]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9eb59f63157b3cff epochNumber=1
{"provingJobId":"1:BASE_PARITY:9eb59f63157b3cff"}
13:13:59 [13:13:59.869]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9eb59f63157b3cff type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9eb59f63157b3cff"}
13:13:59 [13:13:59.870]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9eb59f63157b3cff type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9eb59f63157b3cff"}
13:13:59 [13:13:59.881]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:f53622d9fabd0410 epochNumber=2
{"provingJobId":"2:BASE_PARITY:f53622d9fabd0410"}
13:13:59 [13:13:59.882]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:f53622d9fabd0410 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:f53622d9fabd0410"}
13:13:59 [13:13:59.882]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:f53622d9fabd0410 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:f53622d9fabd0410"}
13:13:59 [13:13:59.943]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:c5b98af264f3dba8 epochNumber=3
{"provingJobId":"3:BASE_PARITY:c5b98af264f3dba8"}
13:13:59 [13:13:59.961]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:14:00 [13:14:00.006]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:d885c29e1f224ce7 epochNumber=4
{"provingJobId":"4:BASE_PARITY:d885c29e1f224ce7"}
13:14:00 [13:14:00.013]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:14:00 [13:14:00.069]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:c4c4d10f804cd6fb epochNumber=5
{"provingJobId":"5:BASE_PARITY:c4c4d10f804cd6fb"}
13:14:00 [13:14:00.090]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:14:00 [13:14:00.125]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.126]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90bf6a6a2b41df02 epochNumber=42
{"provingJobId":"1:BASE_PARITY:90bf6a6a2b41df02"}
13:14:00 [13:14:00.128]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testD8NI5J/42 with map size 1073741824
13:14:00 [13:14:00.133]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.161]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:672f84f07c8d9d04 epochNumber=1
{"provingJobId":"1:BASE_PARITY:672f84f07c8d9d04"}
13:14:00 [13:14:00.170]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.171]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d028e1eef1d0920b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d028e1eef1d0920b"}
13:14:00 [13:14:00.172]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkbz4Um/1 with map size 1073741824
13:14:00 [13:14:00.176]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.198]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:076ee2c6d8ea7132 epochNumber=1
{"provingJobId":"1:BASE_PARITY:076ee2c6d8ea7132"}
13:14:00 [13:14:00.209]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.210]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f04f41e9239deb72 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f04f41e9239deb72"}
13:14:00 [13:14:00.211]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test94M3zr/1 with map size 1073741824
13:14:00 [13:14:00.214]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.233]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f04f41e9239deb72 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f04f41e9239deb72"}
13:14:00 [13:14:00.238]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d2878ef32b3d12f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d2878ef32b3d12f"}
13:14:00 [13:14:00.239]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOZKrIf/1 with map size 1073741824
13:14:00 [13:14:00.241]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.261]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4d2878ef32b3d12f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4d2878ef32b3d12f"}
13:14:00 [13:14:00.263]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4d2878ef32b3d12f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4d2878ef32b3d12f"}
13:14:00 [13:14:00.270]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4d2878ef32b3d12f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4d2878ef32b3d12f"}
13:14:00 [13:14:00.276]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b25cd6ec83df526a epochNumber=1
{"provingJobId":"1:BASE_PARITY:b25cd6ec83df526a"}
13:14:00 [13:14:00.278]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRTTqcp/1 with map size 1073741824
13:14:00 [13:14:00.281]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.308]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.313]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.314]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4713446896e782c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4713446896e782c8"}
13:14:00 [13:14:00.315]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUbSQFl/1 with map size 1073741824
13:14:00 [13:14:00.318]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.337]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4713446896e782c8
{"provingJobId":"1:BASE_PARITY:4713446896e782c8"}
13:14:00 [13:14:00.337]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4713446896e782c8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4713446896e782c8"}
13:14:00 [13:14:00.338]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4713446896e782c8 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4713446896e782c8"}
13:14:00 [13:14:00.349]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.350]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67e0df5a5de78e77 epochNumber=1
{"provingJobId":"1:BASE_PARITY:67e0df5a5de78e77"}
13:14:00 [13:14:00.351]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testimue7Y/1 with map size 1073741824
13:14:00 [13:14:00.354]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.376]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:67e0df5a5de78e77
{"provingJobId":"1:BASE_PARITY:67e0df5a5de78e77"}
13:14:00 [13:14:00.377]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:67e0df5a5de78e77 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:67e0df5a5de78e77"}
13:14:00 [13:14:00.391]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.393]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:23ec62d4f3360479 epochNumber=1
{"provingJobId":"1:BASE_PARITY:23ec62d4f3360479"}
13:14:00 [13:14:00.394]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqZN3UW/1 with map size 1073741824
13:14:00 [13:14:00.402]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.428]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:23ec62d4f3360479 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:23ec62d4f3360479"}
13:14:00 [13:14:00.428]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:23ec62d4f3360479 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:23ec62d4f3360479"}
13:14:00 [13:14:00.440]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.441]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76090fe8c0d05230 epochNumber=1
{"provingJobId":"1:BASE_PARITY:76090fe8c0d05230"}
13:14:00 [13:14:00.441]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testi7gGlR/1 with map size 1073741824
13:14:00 [13:14:00.444]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.465]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:76090fe8c0d05230 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:76090fe8c0d05230"}
13:14:00 [13:14:00.466]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:76090fe8c0d05230 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:76090fe8c0d05230"}
13:14:00 [13:14:00.476]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.477]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2f4c42ba7124a03 epochNumber=0
{"provingJobId":"1:BASE_PARITY:b2f4c42ba7124a03"}
13:14:00 [13:14:00.478]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testr0Svzq/0 with map size 1073741824
13:14:00 [13:14:00.481]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.501]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b2f4c42ba7124a03 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b2f4c42ba7124a03"}
13:14:00 [13:14:00.503]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d91347680418fde epochNumber=0
{"provingJobId":"1:BASE_PARITY:3d91347680418fde"}
13:14:00 [13:14:00.504]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b2f4c42ba7124a03 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b2f4c42ba7124a03"}
13:14:00 [13:14:00.504]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85480f87ca0461ca epochNumber=0
{"provingJobId":"1:BASE_PARITY:85480f87ca0461ca"}
13:14:00 [13:14:00.521]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.524]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.525]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b0c4af6f51adb74 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5b0c4af6f51adb74"}
13:14:00 [13:14:00.526]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfrq2IE/2 with map size 1073741824
13:14:00 [13:14:00.529]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.548]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43818db18b64f3e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43818db18b64f3e2"}
13:14:00 [13:14:00.549]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfrq2IE/1 with map size 1073741824
13:14:00 [13:14:00.554]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.581]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:952fe952cb9d550c epochNumber=1
{"provingJobId":"1:BASE_PARITY:952fe952cb9d550c"}
13:14:00 [13:14:00.584]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4OgBUt/1 with map size 1073741824
13:14:00 [13:14:00.586]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.615]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.616]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6b73e650d2604d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d6b73e650d2604d6"}
13:14:00 [13:14:00.618]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKLYzrZ/1 with map size 1073741824
13:14:00 [13:14:00.623]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.650]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1436ac3f8b746a14 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1436ac3f8b746a14"}
13:14:00 [13:14:00.660]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57aaa33217a74a93 epochNumber=2
{"provingJobId":"1:BASE_PARITY:57aaa33217a74a93"}
13:14:00 [13:14:00.661]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKLYzrZ/2 with map size 1073741824
13:14:00 [13:14:00.663]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a4333d70e3ec27a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a4333d70e3ec27a8"}
13:14:00 [13:14:00.694]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.695]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5a154598c3f7349 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f5a154598c3f7349"}
13:14:00 [13:14:00.696]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2TJ5SP/1 with map size 1073741824
13:14:00 [13:14:00.698]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.716]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b6b3c2de688bf24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b6b3c2de688bf24"}
13:14:00 [13:14:00.721]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:395560314a2c6ed6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:395560314a2c6ed6"}
13:14:00 [13:14:00.722]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2TJ5SP/2 with map size 1073741824
13:14:00 [13:14:00.724]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45901dc7d84c7f1c epochNumber=1
{"provingJobId":"1:BASE_PARITY:45901dc7d84c7f1c"}
13:14:00 [13:14:00.750]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.751]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4635427b66a865e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4635427b66a865e9"}
13:14:00 [13:14:00.752]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4tPwjH/1 with map size 1073741824
13:14:00 [13:14:00.753]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.770]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ce5eec51ad54b83 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0ce5eec51ad54b83"}
13:14:00 [13:14:00.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b266d816d50a576 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2b266d816d50a576"}
13:14:00 [13:14:00.777]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4tPwjH/2 with map size 1073741824
13:14:00 [13:14:00.779]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:db5fce9442e4e7ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:db5fce9442e4e7ff"}
13:14:00 [13:14:00.807]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d36fcf4c88eeb0a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d36fcf4c88eeb0a7"}
13:14:00 [13:14:00.808]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDTv7tB/1 with map size 1073741824
13:14:00 [13:14:00.814]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.836]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6202782ff628bfa epochNumber=1
{"provingJobId":"1:BASE_PARITY:d6202782ff628bfa"}
13:14:00 [13:14:00.841]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d36fcf4c88eeb0a7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d36fcf4c88eeb0a7"}
13:14:00 [13:14:00.853]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.854]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab3b055a6a6fc2bf epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab3b055a6a6fc2bf"}
13:14:00 [13:14:00.854]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testexqMMn/1 with map size 1073741824
13:14:00 [13:14:00.857]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.877]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff876dbe6cbdf0c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff876dbe6cbdf0c3"}
13:14:00 [13:14:00.883]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ab3b055a6a6fc2bf type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:ab3b055a6a6fc2bf"}
13:14:00 [13:14:00.892]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2b87d6eb4d76bb9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d2b87d6eb4d76bb9"}
13:14:00 [13:14:00.895]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTyPAxk/1 with map size 1073741824
13:14:00 [13:14:00.901]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.925]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a77e14b843a7d56 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9a77e14b843a7d56"}
13:14:00 [13:14:00.931]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d2b87d6eb4d76bb9 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:d2b87d6eb4d76bb9"}
13:14:00 [13:14:00.934]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:402369ff6f3b8ff2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:402369ff6f3b8ff2"}
13:14:00 [13:14:00.936]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOk6mCI/1 with map size 1073741824
13:14:00 [13:14:00.942]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:00 [13:14:00.961]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:402369ff6f3b8ff2
{"provingJobId":"1:BASE_PARITY:402369ff6f3b8ff2"}
13:14:00 [13:14:00.961]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:402369ff6f3b8ff2 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:402369ff6f3b8ff2"}
13:14:00 [13:14:00.967]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1aca46587e322410 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1aca46587e322410"}
13:14:00 [13:14:00.973]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:402369ff6f3b8ff2 has already been completed
{"provingJobId":"1:BASE_PARITY:402369ff6f3b8ff2"}
13:14:00 [13:14:00.977]
INFO:
prover-client:proving-broker Proving Broker started
13:14:00 [13:14:00.978]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb4d9a8327852068 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb4d9a8327852068"}
13:14:00 [13:14:00.979]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSvEKTs/1 with map size 1073741824
13:14:00 [13:14:00.983]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:01 [13:14:01.008]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a0d5144e42f803ca epochNumber=2
{"provingJobId":"1:BASE_PARITY:a0d5144e42f803ca"}
13:14:01 [13:14:01.010]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSvEKTs/2 with map size 1073741824
13:14:01 [13:14:01.012]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:02 [13:14:02.081]
INFO:
prover-client:proving-broker Proving Broker started
13:14:02 [13:14:02.088]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fb4d9a8327852068 settled=false
{"provingJobId":"1:BASE_PARITY:fb4d9a8327852068","status":"pending"}
13:14:02 [13:14:02.090]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a0d5144e42f803ca settled=false
{"provingJobId":"1:BASE_PARITY:a0d5144e42f803ca","status":"pending"}
13:14:02 [13:14:02.092]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fb4d9a8327852068 type=BASE_PARITY startedAt=1750166041029 older agent has taken job
{"provingJobId":"1:BASE_PARITY:fb4d9a8327852068"}
13:14:02 [13:14:02.092]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fb4d9a8327852068 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:fb4d9a8327852068"}
13:14:02 [13:14:02.096]
INFO:
prover-client:proving-broker Proving Broker started
13:14:02 [13:14:02.097]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae2b651fa3df8f58 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae2b651fa3df8f58"}
13:14:02 [13:14:02.098]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8xv2Lt/1 with map size 1073741824
13:14:02 [13:14:02.099]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:02 [13:14:02.116]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cbaf147b3be60e0a epochNumber=2
{"provingJobId":"1:BASE_PARITY:cbaf147b3be60e0a"}
13:14:02 [13:14:02.117]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8xv2Lt/2 with map size 1073741824
13:14:02 [13:14:02.120]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:03 [13:14:03.138]
INFO:
prover-client:proving-broker Proving Broker started
13:14:03 [13:14:03.140]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ae2b651fa3df8f58 settled=false
{"provingJobId":"1:BASE_PARITY:ae2b651fa3df8f58","status":"pending"}
13:14:03 [13:14:03.142]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cbaf147b3be60e0a settled=false
{"provingJobId":"1:BASE_PARITY:cbaf147b3be60e0a","status":"pending"}
13:14:03 [13:14:03.143]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ae2b651fa3df8f58 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:ae2b651fa3df8f58"}
13:14:03 [13:14:03.147]
INFO:
prover-client:proving-broker Proving Broker started
13:14:03 [13:14:03.148]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be0b0f4b5975f02e epochNumber=1
{"provingJobId":"1:BASE_PARITY:be0b0f4b5975f02e"}
13:14:03 [13:14:03.149]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQ7VagQ/1 with map size 1073741824
13:14:03 [13:14:03.151]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:03 [13:14:03.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e7f627af5d886e2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7e7f627af5d886e2"}
13:14:03 [13:14:03.169]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQ7VagQ/2 with map size 1073741824
13:14:03 [13:14:03.171]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:04 [13:14:04.187]
INFO:
prover-client:proving-broker Proving Broker started
13:14:04 [13:14:04.190]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:be0b0f4b5975f02e settled=false
{"provingJobId":"1:BASE_PARITY:be0b0f4b5975f02e","status":"pending"}
13:14:04 [13:14:04.192]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7e7f627af5d886e2 settled=false
{"provingJobId":"1:BASE_PARITY:7e7f627af5d886e2","status":"pending"}
13:14:04 [13:14:04.193]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:be0b0f4b5975f02e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:be0b0f4b5975f02e"}
13:14:04 [13:14:04.193]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:be0b0f4b5975f02e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:be0b0f4b5975f02e"}
13:14:04 [13:14:04.205]
INFO:
prover-client:proving-broker Proving Broker started
13:14:04 [13:14:04.205]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:410c9e6e8d84ecf9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:410c9e6e8d84ecf9"}
13:14:04 [13:14:04.208]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoo8T88/1 with map size 1073741824
13:14:04 [13:14:04.211]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:04 [13:14:04.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3baa7ef659acc7e3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3baa7ef659acc7e3"}
13:14:04 [13:14:04.237]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testoo8T88/2 with map size 1073741824
13:14:04 [13:14:04.238]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:04 [13:14:04.256]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:410c9e6e8d84ecf9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:410c9e6e8d84ecf9"}
13:14:04 [13:14:04.262]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3baa7ef659acc7e3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3baa7ef659acc7e3"}
13:14:04 [13:14:04.273]
INFO:
prover-client:proving-broker Proving Broker started
13:14:04 [13:14:04.274]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d40bcf53b061097 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5d40bcf53b061097"}
13:14:04 [13:14:04.274]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSU8384/1 with map size 1073741824
13:14:04 [13:14:04.276]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:04 [13:14:04.294]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2fbc097e7c3572bf epochNumber=2
{"provingJobId":"1:BASE_PARITY:2fbc097e7c3572bf"}
13:14:04 [13:14:04.295]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSU8384/2 with map size 1073741824
13:14:04 [13:14:04.297]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:04 [13:14:04.313]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5d40bcf53b061097 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5d40bcf53b061097"}
13:14:04 [13:14:04.314]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5d40bcf53b061097 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5d40bcf53b061097"}
13:14:04 [13:14:04.321]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2fbc097e7c3572bf type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2fbc097e7c3572bf"}
13:14:04 [13:14:04.331]
INFO:
prover-client:proving-broker Proving Broker started
13:14:04 [13:14:04.332]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7e0c8e25cb19f34c err=test error
{"provingJoId":"1:BASE_PARITY:7e0c8e25cb19f34c"}
13:14:04 [13:14:04.338]
INFO:
prover-client:proving-broker Proving Broker started
13:14:04 [13:14:04.339]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2dc886b956ef8aed not found
{"provingJobId":"1:BASE_PARITY:2dc886b956ef8aed"}
13:14:04 [13:14:04.343]
INFO:
prover-client:proving-broker Proving Broker started
13:14:04 [13:14:04.345]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15deff0be6b45674 epochNumber=1
{"provingJobId":"1:BASE_PARITY:15deff0be6b45674"}
13:14:04 [13:14:04.346]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testA1JKzV/1 with map size 1073741824
13:14:04 [13:14:04.350]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:04 [13:14:04.375]
INFO:
prover-client:proving-broker Proving Broker started
13:14:04 [13:14:04.376]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17743dda3b2b027c epochNumber=1
{"provingJobId":"1:BASE_PARITY:17743dda3b2b027c"}
13:14:04 [13:14:04.377]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testix2keq/1 with map size 1073741824
13:14:04 [13:14:04.379]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:04 [13:14:04.506]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:17743dda3b2b027c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:17743dda3b2b027c"}
13:14:04 [13:14:04.519]
INFO:
prover-client:proving-broker Proving Broker started
13:14:04 [13:14:04.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9b1ae29d2ce8e1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:c9b1ae29d2ce8e1f"}
13:14:04 [13:14:04.522]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test03fXhh/1 with map size 1073741824
13:14:04 [13:14:04.527]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:04 [13:14:04.652]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c9b1ae29d2ce8e1f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c9b1ae29d2ce8e1f"}
13:14:04 [13:14:04.657]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:257bc51268034193 epochNumber=10
{"provingJobId":"1:BASE_PARITY:257bc51268034193"}
13:14:04 [13:14:04.659]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test03fXhh/10 with map size 1073741824
13:14:04 [13:14:04.661]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:04 [13:14:04.704]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:14:04 [13:14:04.704]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:14:04 [13:14:04.812]
INFO:
prover-client:proving-broker Proving Broker started
13:14:04 [13:14:04.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:badcd80797311bfa epochNumber=1
{"provingJobId":"1:BASE_PARITY:badcd80797311bfa"}
13:14:04 [13:14:04.814]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWxlIER/1 with map size 1073741824
13:14:04 [13:14:04.817]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.014]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:badcd80797311bfa timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:badcd80797311bfa"}
13:14:05 [13:14:05.046]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.048]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c47fb7e5b0ad848 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c47fb7e5b0ad848"}
13:14:05 [13:14:05.050]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVJuTN6/1 with map size 1073741824
13:14:05 [13:14:05.053]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.071]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8c47fb7e5b0ad848 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:8c47fb7e5b0ad848"}
13:14:05 [13:14:05.076]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc3272fe51b53cca epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc3272fe51b53cca"}
13:14:05 [13:14:05.078]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpT4Aiq/1 with map size 1073741824
13:14:05 [13:14:05.080]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.100]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cc3272fe51b53cca type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:cc3272fe51b53cca"}
13:14:05 [13:14:05.101]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cc3272fe51b53cca type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:cc3272fe51b53cca"}
13:14:05 [13:14:05.112]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.113]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff8008717862e666 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff8008717862e666"}
13:14:05 [13:14:05.113]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2THkN7/1 with map size 1073741824
13:14:05 [13:14:05.116]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.134]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ff8008717862e666 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ff8008717862e666"}
13:14:05 [13:14:05.143]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.144]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:deb17148bbff6f87 epochNumber=1
{"provingJobId":"1:BASE_PARITY:deb17148bbff6f87"}
13:14:05 [13:14:05.146]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBXfyEs/1 with map size 1073741824
13:14:05 [13:14:05.148]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.169]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:deb17148bbff6f87 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:deb17148bbff6f87"}
13:14:05 [13:14:05.170]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:737d7d0e1598d205 epochNumber=3
{"provingJobId":"1:BASE_PARITY:737d7d0e1598d205"}
13:14:05 [13:14:05.171]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testBXfyEs/3 with map size 1073741824
13:14:05 [13:14:05.173]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.222]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:14:05 [13:14:05.222]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:14:05 [13:14:05.225]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:deb17148bbff6f87 err=test error
{"provingJoId":"1:BASE_PARITY:deb17148bbff6f87"}
13:14:05 [13:14:05.233]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkDsrOq/1 with map size 1073741824
13:14:05 [13:14:05.235]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.264]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkDsrOq/2 with map size 1073741824
13:14:05 [13:14:05.269]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.290]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.293]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cd9a7027e1d60b44 settled=false
{"provingJobId":"1:BASE_PARITY:cd9a7027e1d60b44","status":"pending"}
13:14:05 [13:14:05.294]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3422b3a34a75cb76 settled=false
{"provingJobId":"1:BASE_PARITY:3422b3a34a75cb76","status":"pending"}
13:14:05 [13:14:05.302]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGXO1XL/1 with map size 1073741824
13:14:05 [13:14:05.305]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.324]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGXO1XL/2 with map size 1073741824
13:14:05 [13:14:05.327]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.365]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.368]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9b789bdb08d7e3ec settled=true
{"provingJobId":"1:BASE_PARITY:9b789bdb08d7e3ec","status":"fulfilled"}
13:14:05 [13:14:05.370]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:6be7abf3270ed8ba settled=true
{"provingJobId":"2:BASE_PARITY:6be7abf3270ed8ba","status":"fulfilled"}
13:14:05 [13:14:05.378]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testB1Jv1s/1 with map size 1073741824
13:14:05 [13:14:05.387]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.417]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testB1Jv1s/2 with map size 1073741824
13:14:05 [13:14:05.419]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.436]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.438]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0dfa85a97d6a6deb settled=true
{"provingJobId":"1:BASE_PARITY:0dfa85a97d6a6deb","status":"fulfilled"}
13:14:05 [13:14:05.440]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c368907c7c675024 settled=false
{"provingJobId":"1:BASE_PARITY:c368907c7c675024","status":"pending"}
13:14:05 [13:14:05.448]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.449]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9b59aad490c6d23 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c9b59aad490c6d23"}
13:14:05 [13:14:05.452]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVqNLXr/1 with map size 1073741824
13:14:05 [13:14:05.456]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.486]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.487]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7a27aac454a2b5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:d7a27aac454a2b5d"}
13:14:05 [13:14:05.488]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:d7a27aac454a2b5d: Error: db error: Error: db error
13:14:05 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:d7a27aac454a2b5d"}
13:14:05 [13:14:05.495]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.496]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24d0ad233c7f3125 epochNumber=1
{"provingJobId":"1:BASE_PARITY:24d0ad233c7f3125"}
13:14:05 [13:14:05.497]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNwyi1J/1 with map size 1073741824
13:14:05 [13:14:05.499]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.518]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:24d0ad233c7f3125 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:24d0ad233c7f3125"}
13:14:05 [13:14:05.519]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:24d0ad233c7f3125 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:24d0ad233c7f3125"}
13:14:05 [13:14:05.531]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.531]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:74b53473ffb4d08b epochNumber=1
{"provingJobId":"1:BASE_PARITY:74b53473ffb4d08b"}
13:14:05 [13:14:05.532]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUwRp4l/1 with map size 1073741824
13:14:05 [13:14:05.536]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.560]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:74b53473ffb4d08b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:74b53473ffb4d08b"}
13:14:05 [13:14:05.561]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:74b53473ffb4d08b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:74b53473ffb4d08b"}
13:14:05 [13:14:05.562]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:74b53473ffb4d08b: Error: db error
13:14:05 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:74b53473ffb4d08b"}
13:14:05 [13:14:05.566]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.567]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54add489eb20df9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:54add489eb20df9f"}
13:14:05 [13:14:05.568]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeoAEpm/1 with map size 1073741824
13:14:05 [13:14:05.573]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.594]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:54add489eb20df9f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:54add489eb20df9f"}
13:14:05 [13:14:05.595]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:54add489eb20df9f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:54add489eb20df9f"}
13:14:05 [13:14:05.608]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58a2af3c0a5db856 epochNumber=1
{"provingJobId":"1:BASE_PARITY:58a2af3c0a5db856"}
13:14:05 [13:14:05.609]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test15KMmR/1 with map size 1073741824
13:14:05 [13:14:05.612]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.634]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:58a2af3c0a5db856 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:58a2af3c0a5db856"}
13:14:05 [13:14:05.634]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:58a2af3c0a5db856 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:58a2af3c0a5db856"}
13:14:05 [13:14:05.635]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:58a2af3c0a5db856 jobErr=test error: Error: db error
13:14:05 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:58a2af3c0a5db856"}
13:14:05 [13:14:05.640]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.641]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d2fb1a2c163cbb44 not found
{"provingJobId":"1:BASE_PARITY:d2fb1a2c163cbb44"}
13:14:05 [13:14:05.644]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.645]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:27117cfdb4f61ced err=test error
{"provingJoId":"1:BASE_PARITY:27117cfdb4f61ced"}
13:14:05 [13:14:05.651]
INFO:
prover-client:proving-broker Proving Broker started
13:14:05 [13:14:05.662]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c58f18cd9355020 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c58f18cd9355020"}
13:14:05 [13:14:05.663]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxFjJN8/1 with map size 1073741824
13:14:05 [13:14:05.666]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.688]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6c58f18cd9355020 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6c58f18cd9355020"}
13:14:05 [13:14:05.689]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6c58f18cd9355020 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6c58f18cd9355020"}
13:14:05 [13:14:05.707]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:a9db3d16e2a4d90e epochNumber=2
{"provingJobId":"2:BASE_PARITY:a9db3d16e2a4d90e"}
13:14:05 [13:14:05.708]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testxFjJN8/2 with map size 1073741824
13:14:05 [13:14:05.713]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.736]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:a9db3d16e2a4d90e type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:a9db3d16e2a4d90e"}
13:14:05 [13:14:05.737]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:a9db3d16e2a4d90e type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:a9db3d16e2a4d90e"}
13:14:05 [13:14:05.803]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:129493470fab835d epochNumber=3
{"provingJobId":"3:BASE_PARITY:129493470fab835d"}
13:14:05 [13:14:05.804]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testxFjJN8/3 with map size 1073741824
13:14:05 [13:14:05.808]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.856]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:14:05 [13:14:05.857]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:14:05 [13:14:05.892]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:60ea5c5fc2916aeb epochNumber=4
{"provingJobId":"4:BASE_PARITY:60ea5c5fc2916aeb"}
13:14:05 [13:14:05.893]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testxFjJN8/4 with map size 1073741824
13:14:05 [13:14:05.897]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:05 [13:14:05.936]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:14:05 [13:14:05.937]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
13:14:05 [13:14:05.980]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:7f9a6505f0ad7fbe epochNumber=5
{"provingJobId":"5:BASE_PARITY:7f9a6505f0ad7fbe"}
13:14:05 [13:14:05.982]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testxFjJN8/5 with map size 1073741824
13:14:05 [13:14:05.987]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:14:06 [13:14:06.016]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:14:06 [13:14:06.016]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
13:14:06
PASS src/proving_broker/proving_broker.test.ts (
13.755 s)
13:14:06 ProvingBroker
13:14:06 Producer API
13:14:06
✓ refuses stale jobs (58 ms)
13:14:06
✓ enqueues jobs (4 ms)
13:14:06
✓ ignores duplicate jobs (4 ms)
13:14:06
✓ reports correct status when enqueuing repeat jobs (5 ms)
13:14:06
✓ throws an error in case of duplicate job IDs (32 ms)
13:14:06
✓ returns not-found status for non-existing jobs (2 ms)
13:14:06
✓ cancels jobs in queue (4 ms)
13:14:06
✓ cancels jobs in-progress (3 ms)
13:14:06
✓ returns job result if successful (3 ms)
13:14:06
✓ returns job error if failed (2 ms)
13:14:06
✓ correctly returns job status for concurrent writes (3 ms)
13:14:06
✓ refuses stale jobs (45 ms)
13:14:06
✓ enqueues jobs (39 ms)
13:14:06
✓ ignores duplicate jobs (28 ms)
13:14:06
✓ reports correct status when enqueuing repeat jobs (36 ms)
13:14:06
✓ throws an error in case of duplicate job IDs (31 ms)
13:14:06
✓ returns not-found status for non-existing jobs (6 ms)
13:14:06
✓ cancels jobs in queue (36 ms)
13:14:06
✓ cancels jobs in-progress (39 ms)
13:14:06
✓ returns job result if successful (51 ms)
13:14:06
✓ returns job error if failed (36 ms)
13:14:06
✓ correctly returns job status for concurrent writes (43 ms)
13:14:06 Consumer API
13:14:06
✓ returns undefined if no jobs are available (22 ms)
13:14:06
✓ returns jobs in priority order (3 ms)
13:14:06
✓ returns undefined if no jobs are available for the given allowList (1 ms)
13:14:06
✓ returns a job if it is in the allowList (2 ms)
13:14:06
✓ returns the most important job if it is in the allowList (2 ms)
13:14:06
✓ returns any job if filter is empty (3 ms)
13:14:06
✓ returns a new job when reporting job success (2 ms)
13:14:06
✓ returns a new job when reporting permanent error (2 ms)
13:14:06
✓ returns a new job when reporting retry-able error (2 ms)
13:14:06
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
13:14:06
✓ returns a new job if job is already in progress elsewhere (1060 ms)
13:14:06
✓ avoids sending the same job to a new agent after a restart (1006 ms)
13:14:06
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
13:14:06
✓ tracks job result if in progress (4 ms)
13:14:06
✓ tracks job result even if job is in queue (3 ms)
13:14:06
✓ ignores reported job error if unknown job (2 ms)
13:14:06
✓ ignores job result if unknown job (1 ms)
13:14:06
✓ returns undefined if no jobs are available (3 ms)
13:14:06
✓ returns jobs in priority order (56 ms)
13:14:06
✓ returns undefined if no jobs are available for the given allowList (34 ms)
13:14:06
✓ returns a job if it is in the allowList (78 ms)
13:14:06
✓ returns the most important job if it is in the allowList (56 ms)
13:14:06
✓ returns any job if filter is empty (57 ms)
13:14:06
✓ returns a new job when reporting job success (44 ms)
13:14:06
✓ returns a new job when reporting permanent error (41 ms)
13:14:06
✓ returns a new job when reporting retry-able error (43 ms)
13:14:06
✓ returns a new job when reporting progress if current one is cancelled (41 ms)
13:14:06
✓ returns a new job if job is already in progress elsewhere (1119 ms)
13:14:06
✓ avoids sending the same job to a new agent after a restart (1050 ms)
13:14:06
✓ avoids sending a completed job to a new agent after a restart (1058 ms)
13:14:06
✓ tracks job result if in progress (68 ms)
13:14:06
✓ tracks job result even if job is in queue (56 ms)
13:14:06
✓ ignores reported job error if unknown job (5 ms)
13:14:06
✓ ignores job result if unknown job (6 ms)
13:14:06 Timeouts
13:14:06
✓ tracks in progress jobs (2 ms)
13:14:06
✓ re-enqueues jobs that time out (110 ms)
13:14:06
✓ cancel stale jobs that time out (232 ms)
13:14:06
✓ keeps the jobs in progress while it is alive (205 ms)
13:14:06
✓ tracks in progress jobs (30 ms)
13:14:06
✓ re-enqueues jobs that time out (141 ms)
13:14:06
✓ cancel stale jobs that time out (294 ms)
13:14:06
✓ keeps the jobs in progress while it is alive (236 ms)
13:14:06 Retries
13:14:06
✓ retries jobs (4 ms)
13:14:06
✓ retries up to a maximum number of times (3 ms)
13:14:06
✓ passing retry=false does not retry (2 ms)
13:14:06
✓ does not retry if job is stale (30 ms)
13:14:06
✓ retries jobs (31 ms)
13:14:06
✓ retries up to a maximum number of times (34 ms)
13:14:06
✓ passing retry=false does not retry (31 ms)
13:14:06
✓ does not retry if job is stale (85 ms)
13:14:06 Database management
13:14:06
✓ re-enqueues proof requests on start (4 ms)
13:14:06
✓ restores proof results on start (2 ms)
13:14:06
✓ only re-enqueues unfinished jobs (2 ms)
13:14:06
✓ saves job when enqueued (5 ms)
13:14:06
✓ does not retain job if database fails to save (4 ms)
13:14:06
✓ saves job result (3 ms)
13:14:06
✓ saves result even if database fails to save (3 ms)
13:14:06
✓ saves job error (4 ms)
13:14:06
✓ saves job error even if database fails to save (8 ms)
13:14:06
✓ does not save job result if job is unknown (3 ms)
13:14:06
✓ does not save job error if job is unknown (3 ms)
13:14:06
✓ cleans up old jobs periodically (264 ms)
13:14:06
✓ re-enqueues proof requests on start (71 ms)
13:14:06
✓ restores proof results on start (74 ms)
13:14:06
✓ only re-enqueues unfinished jobs (70 ms)
13:14:06
✓ saves job when enqueued (40 ms)
13:14:06
✓ does not retain job if database fails to save (6 ms)
13:14:06
✓ saves job result (35 ms)
13:14:06
✓ saves result even if database fails to save (37 ms)
13:14:06
✓ saves job error (41 ms)
13:14:06
✓ saves job error even if database fails to save (32 ms)
13:14:06
✓ does not save job result if job is unknown (5 ms)
13:14:06
✓ does not save job error if job is unknown (3 ms)
13:14:06
✓ cleans up old jobs periodically (415 ms)
13:14:06
13:14:06
Test Suites: 1 passed, 1 total
13:14:06
Tests: 96 passed, 96 total
13:14:06
Snapshots: 0 total
13:14:06
Time: 13.863 s
13:14:06
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
13:14:06
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?