Parent Log:
http://ci.aztec-labs.com/51fad2508877a18c
Command: 2f76ae4bba915073 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/87c4b689268afecc86ab007c3587e45110daa43d
Env: REF_NAME=gh-readonly-queue/next/pr-15220-282da3bd25d49b92970e6fc85f110092e8566299 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 24 16:34:56 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15220_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
16:35:00 [16:35:00.804]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.812]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04149306209bdd93 epochNumber=42
{"provingJobId":"1:BASE_PARITY:04149306209bdd93"}
16:35:00 [16:35:00.815]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:a9b64155292261d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a9b64155292261d9"}
16:35:00 [16:35:00.851]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.852]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4711f3642c63ad25 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4711f3642c63ad25"}
16:35:00 [16:35:00.853]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:efe45faac9ec3dea epochNumber=1
{"provingJobId":"1:BASE_PARITY:efe45faac9ec3dea"}
16:35:00 [16:35:00.855]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.856]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e807cef8d1be6f7b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e807cef8d1be6f7b"}
16:35:00 [16:35:00.857]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e807cef8d1be6f7b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e807cef8d1be6f7b"}
16:35:00 [16:35:00.859]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.860]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb3f087ce408fd8d epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb3f087ce408fd8d"}
16:35:00 [16:35:00.861]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cb3f087ce408fd8d epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:cb3f087ce408fd8d"}
16:35:00 [16:35:00.862]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cb3f087ce408fd8d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cb3f087ce408fd8d"}
16:35:00 [16:35:00.862]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cb3f087ce408fd8d epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:cb3f087ce408fd8d"}
16:35:00 [16:35:00.864]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.864]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61331a4a2b15f2e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:61331a4a2b15f2e3"}
16:35:00 [16:35:00.895]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.896]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.897]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cfd8a5a544cb6aca epochNumber=1
{"provingJobId":"1:BASE_PARITY:cfd8a5a544cb6aca"}
16:35:00 [16:35:00.898]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:cfd8a5a544cb6aca
{"provingJobId":"1:BASE_PARITY:cfd8a5a544cb6aca"}
16:35:00 [16:35:00.898]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cfd8a5a544cb6aca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:cfd8a5a544cb6aca"}
16:35:00 [16:35:00.899]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cfd8a5a544cb6aca type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:cfd8a5a544cb6aca"}
16:35:00 [16:35:00.900]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.901]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1573214a6815757f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1573214a6815757f"}
16:35:00 [16:35:00.902]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1573214a6815757f
{"provingJobId":"1:BASE_PARITY:1573214a6815757f"}
16:35:00 [16:35:00.902]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1573214a6815757f type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1573214a6815757f"}
16:35:00 [16:35:00.904]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.904]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab394a35b18834f6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab394a35b18834f6"}
16:35:00 [16:35:00.905]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ab394a35b18834f6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ab394a35b18834f6"}
16:35:00 [16:35:00.905]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ab394a35b18834f6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ab394a35b18834f6"}
16:35:00 [16:35:00.907]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.907]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:173101add018bac0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:173101add018bac0"}
16:35:00 [16:35:00.908]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:173101add018bac0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:173101add018bac0"}
16:35:00 [16:35:00.908]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:173101add018bac0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:173101add018bac0"}
16:35:00 [16:35:00.909]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.910]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cffb3caf509d6c75 epochNumber=0
{"provingJobId":"1:BASE_PARITY:cffb3caf509d6c75"}
16:35:00 [16:35:00.910]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cffb3caf509d6c75 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:cffb3caf509d6c75"}
16:35:00 [16:35:00.911]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:07af23ae6cb02221 epochNumber=0
{"provingJobId":"1:BASE_PARITY:07af23ae6cb02221"}
16:35:00 [16:35:00.911]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cffb3caf509d6c75 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:cffb3caf509d6c75"}
16:35:00 [16:35:00.911]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e7f099c42339fc9 epochNumber=0
{"provingJobId":"1:BASE_PARITY:1e7f099c42339fc9"}
16:35:00 [16:35:00.913]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.922]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.922]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b1cbf326c8200ec1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b1cbf326c8200ec1"}
16:35:00 [16:35:00.923]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0bdc57ad67a23874 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0bdc57ad67a23874"}
16:35:00 [16:35:00.924]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.925]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e18beb9beda248fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:e18beb9beda248fd"}
16:35:00 [16:35:00.926]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.927]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13fe8e58aff433e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13fe8e58aff433e0"}
16:35:00 [16:35:00.927]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e07acf77cc07c294 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e07acf77cc07c294"}
16:35:00 [16:35:00.928]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85ed1bca1a55bfac epochNumber=2
{"provingJobId":"1:BASE_PARITY:85ed1bca1a55bfac"}
16:35:00 [16:35:00.928]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:39af0c08f432d3d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:39af0c08f432d3d0"}
16:35:00 [16:35:00.929]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.930]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4682e0c692868b1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4682e0c692868b1f"}
16:35:00 [16:35:00.930]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47113856cefcb9dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:47113856cefcb9dc"}
16:35:00 [16:35:00.931]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1765caa5460174a8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1765caa5460174a8"}
16:35:00 [16:35:00.931]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53f63b47aa380a77 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53f63b47aa380a77"}
16:35:00 [16:35:00.932]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.933]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1412d45d6cfeaa04 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1412d45d6cfeaa04"}
16:35:00 [16:35:00.933]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9bfd1e4977ff6d84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9bfd1e4977ff6d84"}
16:35:00 [16:35:00.933]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6261b5ba2f2106aa epochNumber=2
{"provingJobId":"1:BASE_PARITY:6261b5ba2f2106aa"}
16:35:00 [16:35:00.934]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fab1b23b81493329 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fab1b23b81493329"}
16:35:00 [16:35:00.935]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5698620e86e39e57 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5698620e86e39e57"}
16:35:00 [16:35:00.936]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6af986542516264a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6af986542516264a"}
16:35:00 [16:35:00.936]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5698620e86e39e57 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5698620e86e39e57"}
16:35:00 [16:35:00.938]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.938]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72b61918ec27dc26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:72b61918ec27dc26"}
16:35:00 [16:35:00.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4342ead8dbd5520e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4342ead8dbd5520e"}
16:35:00 [16:35:00.939]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:72b61918ec27dc26 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:72b61918ec27dc26"}
16:35:00 [16:35:00.940]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.941]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40e3a40a632168b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:40e3a40a632168b6"}
16:35:00 [16:35:00.941]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:368007834b9782c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:368007834b9782c5"}
16:35:00 [16:35:00.941]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:40e3a40a632168b6 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:40e3a40a632168b6"}
16:35:00 [16:35:00.943]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc0141d54c5d00df epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc0141d54c5d00df"}
16:35:00 [16:35:00.944]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:cc0141d54c5d00df
{"provingJobId":"1:BASE_PARITY:cc0141d54c5d00df"}
16:35:00 [16:35:00.944]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cc0141d54c5d00df type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:cc0141d54c5d00df"}
16:35:00 [16:35:00.945]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7b143c2d7bc2270 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7b143c2d7bc2270"}
16:35:00 [16:35:00.945]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cc0141d54c5d00df has already been completed
{"provingJobId":"1:BASE_PARITY:cc0141d54c5d00df"}
16:35:00 [16:35:00.947]
INFO:
prover-client:proving-broker Proving Broker started
16:35:00 [16:35:00.947]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2912defcb86ef08 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e2912defcb86ef08"}
16:35:00 [16:35:00.947]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c06c65aad8ac6e3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6c06c65aad8ac6e3"}
16:35:02 [16:35:02.000]
INFO:
prover-client:proving-broker Proving Broker started
16:35:02 [16:35:02.000]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e2912defcb86ef08 settled=false
{"provingJobId":"1:BASE_PARITY:e2912defcb86ef08","status":"pending"}
16:35:02 [16:35:02.001]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6c06c65aad8ac6e3 settled=false
{"provingJobId":"1:BASE_PARITY:6c06c65aad8ac6e3","status":"pending"}
16:35:02 [16:35:02.002]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e2912defcb86ef08 type=BASE_PARITY startedAt=1750782900948 older agent has taken job
{"provingJobId":"1:BASE_PARITY:e2912defcb86ef08"}
16:35:02 [16:35:02.002]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e2912defcb86ef08 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:e2912defcb86ef08"}
16:35:02 [16:35:02.003]
INFO:
prover-client:proving-broker Proving Broker started
16:35:02 [16:35:02.004]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:276406d1b061a39a epochNumber=1
{"provingJobId":"1:BASE_PARITY:276406d1b061a39a"}
16:35:02 [16:35:02.004]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71b12de81e38d237 epochNumber=2
{"provingJobId":"1:BASE_PARITY:71b12de81e38d237"}
16:35:03 [16:35:03.005]
INFO:
prover-client:proving-broker Proving Broker started
16:35:03 [16:35:03.005]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:276406d1b061a39a settled=false
{"provingJobId":"1:BASE_PARITY:276406d1b061a39a","status":"pending"}
16:35:03 [16:35:03.006]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:71b12de81e38d237 settled=false
{"provingJobId":"1:BASE_PARITY:71b12de81e38d237","status":"pending"}
16:35:03 [16:35:03.006]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:276406d1b061a39a type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:276406d1b061a39a"}
16:35:03 [16:35:03.008]
INFO:
prover-client:proving-broker Proving Broker started
16:35:03 [16:35:03.008]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:318d0994fe03593f epochNumber=1
{"provingJobId":"1:BASE_PARITY:318d0994fe03593f"}
16:35:03 [16:35:03.009]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57aee3c861bd1a5d epochNumber=2
{"provingJobId":"1:BASE_PARITY:57aee3c861bd1a5d"}
16:35:04 [16:35:04.009]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.010]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:318d0994fe03593f settled=false
{"provingJobId":"1:BASE_PARITY:318d0994fe03593f","status":"pending"}
16:35:04 [16:35:04.010]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:57aee3c861bd1a5d settled=false
{"provingJobId":"1:BASE_PARITY:57aee3c861bd1a5d","status":"pending"}
16:35:04 [16:35:04.011]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:318d0994fe03593f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:318d0994fe03593f"}
16:35:04 [16:35:04.011]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:318d0994fe03593f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:318d0994fe03593f"}
16:35:04 [16:35:04.013]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.013]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e92ba72432220b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e92ba72432220b3"}
16:35:04 [16:35:04.014]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b1a993618dc9b87 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4b1a993618dc9b87"}
16:35:04 [16:35:04.014]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4e92ba72432220b3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4e92ba72432220b3"}
16:35:04 [16:35:04.015]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4b1a993618dc9b87 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4b1a993618dc9b87"}
16:35:04 [16:35:04.016]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.017]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb728704dfc7d6d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb728704dfc7d6d6"}
16:35:04 [16:35:04.017]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e55aa4419ad4902b epochNumber=2
{"provingJobId":"1:BASE_PARITY:e55aa4419ad4902b"}
16:35:04 [16:35:04.017]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb728704dfc7d6d6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:eb728704dfc7d6d6"}
16:35:04 [16:35:04.017]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:eb728704dfc7d6d6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:eb728704dfc7d6d6"}
16:35:04 [16:35:04.018]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e55aa4419ad4902b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e55aa4419ad4902b"}
16:35:04 [16:35:04.019]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.019]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:99c9014169546154 err=test error
{"provingJoId":"1:BASE_PARITY:99c9014169546154"}
16:35:04 [16:35:04.020]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.021]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:41d6ec7dce9458b7 not found
{"provingJobId":"1:BASE_PARITY:41d6ec7dce9458b7"}
16:35:04 [16:35:04.022]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.023]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8d6ad958d14e545 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8d6ad958d14e545"}
16:35:04 [16:35:04.024]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:550b4065cb18311f epochNumber=1
{"provingJobId":"1:BASE_PARITY:550b4065cb18311f"}
16:35:04 [16:35:04.127]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:550b4065cb18311f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:550b4065cb18311f"}
16:35:04 [16:35:04.134]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.135]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ed55797688ddeb5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ed55797688ddeb5"}
16:35:04 [16:35:04.236]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9ed55797688ddeb5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9ed55797688ddeb5"}
16:35:04 [16:35:04.244]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cdaab434facadde8 epochNumber=10
{"provingJobId":"1:BASE_PARITY:cdaab434facadde8"}
16:35:04 [16:35:04.262]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:04 [16:35:04.373]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.374]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10a231d83704a51a epochNumber=1
{"provingJobId":"1:BASE_PARITY:10a231d83704a51a"}
16:35:04 [16:35:04.528]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:10a231d83704a51a timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:10a231d83704a51a"}
16:35:04 [16:35:04.578]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.579]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e360e2c31f1c0493 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e360e2c31f1c0493"}
16:35:04 [16:35:04.580]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e360e2c31f1c0493 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:e360e2c31f1c0493"}
16:35:04 [16:35:04.581]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.581]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f8faac98b693acd epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f8faac98b693acd"}
16:35:04 [16:35:04.582]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5f8faac98b693acd type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:5f8faac98b693acd"}
16:35:04 [16:35:04.582]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5f8faac98b693acd type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:5f8faac98b693acd"}
16:35:04 [16:35:04.583]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.584]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c38c38b5a246762 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c38c38b5a246762"}
16:35:04 [16:35:04.584]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6c38c38b5a246762 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6c38c38b5a246762"}
16:35:04 [16:35:04.585]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.586]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b49bf2b5dfa7aa50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b49bf2b5dfa7aa50"}
16:35:04 [16:35:04.586]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b49bf2b5dfa7aa50 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b49bf2b5dfa7aa50"}
16:35:04 [16:35:04.587]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2bc00d7a3db5e88 epochNumber=3
{"provingJobId":"1:BASE_PARITY:d2bc00d7a3db5e88"}
16:35:04 [16:35:04.611]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:04 [16:35:04.612]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b49bf2b5dfa7aa50 err=test error
{"provingJoId":"1:BASE_PARITY:b49bf2b5dfa7aa50"}
16:35:04 [16:35:04.614]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.614]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1deefe07bbccdaae settled=false
{"provingJobId":"1:BASE_PARITY:1deefe07bbccdaae","status":"pending"}
16:35:04 [16:35:04.615]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:33928b4405934b40 settled=false
{"provingJobId":"1:BASE_PARITY:33928b4405934b40","status":"pending"}
16:35:04 [16:35:04.618]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.618]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c00476e22579d77a settled=true
{"provingJobId":"1:BASE_PARITY:c00476e22579d77a","status":"fulfilled"}
16:35:04 [16:35:04.618]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:53d9a1ecbcfe50b2 settled=true
{"provingJobId":"2:BASE_PARITY:53d9a1ecbcfe50b2","status":"fulfilled"}
16:35:04 [16:35:04.620]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.620]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9cafaa7d77047dbb settled=true
{"provingJobId":"1:BASE_PARITY:9cafaa7d77047dbb","status":"fulfilled"}
16:35:04 [16:35:04.620]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1f23bb7f5445c66c settled=false
{"provingJobId":"1:BASE_PARITY:1f23bb7f5445c66c","status":"pending"}
16:35:04 [16:35:04.622]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.623]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d017cb99b177a200 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d017cb99b177a200"}
16:35:04 [16:35:04.625]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.625]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26a73efea8eb4ec4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:26a73efea8eb4ec4"}
16:35:04 [16:35:04.626]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:26a73efea8eb4ec4: Error: db error: Error: db error
16:35:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:26a73efea8eb4ec4"}
16:35:04 [16:35:04.627]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.628]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4deaf82565cbfbb6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4deaf82565cbfbb6"}
16:35:04 [16:35:04.628]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4deaf82565cbfbb6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4deaf82565cbfbb6"}
16:35:04 [16:35:04.629]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4deaf82565cbfbb6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4deaf82565cbfbb6"}
16:35:04 [16:35:04.630]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.630]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:538566f0cbb3581e epochNumber=1
{"provingJobId":"1:BASE_PARITY:538566f0cbb3581e"}
16:35:04 [16:35:04.631]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:538566f0cbb3581e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:538566f0cbb3581e"}
16:35:04 [16:35:04.631]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:538566f0cbb3581e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:538566f0cbb3581e"}
16:35:04 [16:35:04.631]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:538566f0cbb3581e: Error: db error
16:35:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:538566f0cbb3581e"}
16:35:04 [16:35:04.633]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.633]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6344d45db63c489 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b6344d45db63c489"}
16:35:04 [16:35:04.634]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b6344d45db63c489 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b6344d45db63c489"}
16:35:04 [16:35:04.634]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b6344d45db63c489 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b6344d45db63c489"}
16:35:04 [16:35:04.635]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.636]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e68ff6b4dec83e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e68ff6b4dec83e5"}
16:35:04 [16:35:04.636]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3e68ff6b4dec83e5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3e68ff6b4dec83e5"}
16:35:04 [16:35:04.636]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3e68ff6b4dec83e5 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3e68ff6b4dec83e5"}
16:35:04 [16:35:04.637]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:3e68ff6b4dec83e5 jobErr=test error: Error: db error
16:35:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:3e68ff6b4dec83e5"}
16:35:04 [16:35:04.638]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.638]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0a5eec639c3238e5 not found
{"provingJobId":"1:BASE_PARITY:0a5eec639c3238e5"}
16:35:04 [16:35:04.640]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.640]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1c9ea5e9a986cf22 err=test error
{"provingJoId":"1:BASE_PARITY:1c9ea5e9a986cf22"}
16:35:04 [16:35:04.641]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.652]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:093213d41903ff1d epochNumber=1
{"provingJobId":"1:BASE_PARITY:093213d41903ff1d"}
16:35:04 [16:35:04.653]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:093213d41903ff1d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:093213d41903ff1d"}
16:35:04 [16:35:04.654]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:093213d41903ff1d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:093213d41903ff1d"}
16:35:04 [16:35:04.666]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:6a75055060503df5 epochNumber=2
{"provingJobId":"2:BASE_PARITY:6a75055060503df5"}
16:35:04 [16:35:04.667]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:6a75055060503df5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:6a75055060503df5"}
16:35:04 [16:35:04.668]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:6a75055060503df5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:6a75055060503df5"}
16:35:04 [16:35:04.730]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:c3b29819b9ee7594 epochNumber=3
{"provingJobId":"3:BASE_PARITY:c3b29819b9ee7594"}
16:35:04 [16:35:04.745]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:04 [16:35:04.792]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:28ceda6da20efbc6 epochNumber=4
{"provingJobId":"4:BASE_PARITY:28ceda6da20efbc6"}
16:35:04 [16:35:04.797]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:04 [16:35:04.855]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:d5079453c5c0cb49 epochNumber=5
{"provingJobId":"5:BASE_PARITY:d5079453c5c0cb49"}
16:35:04 [16:35:04.874]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:04 [16:35:04.911]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.912]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94284789b2feb30b epochNumber=42
{"provingJobId":"1:BASE_PARITY:94284789b2feb30b"}
16:35:04 [16:35:04.913]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testaklBHL/42 with map size 1073741824
16:35:04 [16:35:04.918]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:04 [16:35:04.946]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:fec97797d30f38d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fec97797d30f38d5"}
16:35:04 [16:35:04.952]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.953]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f61b9f9b14e6292 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f61b9f9b14e6292"}
16:35:04 [16:35:04.955]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmS5glK/1 with map size 1073741824
16:35:04 [16:35:04.957]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:04 [16:35:04.981]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:27d8e34ba0be1518 epochNumber=1
{"provingJobId":"1:BASE_PARITY:27d8e34ba0be1518"}
16:35:04 [16:35:04.994]
INFO:
prover-client:proving-broker Proving Broker started
16:35:04 [16:35:04.995]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3ed7da7a27014c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3ed7da7a27014c7"}
16:35:04 [16:35:04.995]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNmVoYP/1 with map size 1073741824
16:35:04 [16:35:04.997]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.017]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a3ed7da7a27014c7 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:a3ed7da7a27014c7"}
16:35:05 [16:35:05.021]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.022]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f3742ebb0e39463 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f3742ebb0e39463"}
16:35:05 [16:35:05.025]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDhw4Cp/1 with map size 1073741824
16:35:05 [16:35:05.027]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.049]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5f3742ebb0e39463 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5f3742ebb0e39463"}
16:35:05 [16:35:05.050]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5f3742ebb0e39463 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5f3742ebb0e39463"}
16:35:05 [16:35:05.057]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5f3742ebb0e39463 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5f3742ebb0e39463"}
16:35:05 [16:35:05.063]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.064]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d32fa1c8eb418b52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d32fa1c8eb418b52"}
16:35:05 [16:35:05.065]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuUeLpJ/1 with map size 1073741824
16:35:05 [16:35:05.067]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.090]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.093]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.094]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e6aa60f475b64bc epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e6aa60f475b64bc"}
16:35:05 [16:35:05.095]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCC6bUV/1 with map size 1073741824
16:35:05 [16:35:05.097]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.115]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5e6aa60f475b64bc
{"provingJobId":"1:BASE_PARITY:5e6aa60f475b64bc"}
16:35:05 [16:35:05.116]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5e6aa60f475b64bc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5e6aa60f475b64bc"}
16:35:05 [16:35:05.116]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5e6aa60f475b64bc type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5e6aa60f475b64bc"}
16:35:05 [16:35:05.126]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:600510a0a79f8813 epochNumber=1
{"provingJobId":"1:BASE_PARITY:600510a0a79f8813"}
16:35:05 [16:35:05.128]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpWLSui/1 with map size 1073741824
16:35:05 [16:35:05.130]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.149]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:600510a0a79f8813
{"provingJobId":"1:BASE_PARITY:600510a0a79f8813"}
16:35:05 [16:35:05.149]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:600510a0a79f8813 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:600510a0a79f8813"}
16:35:05 [16:35:05.161]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.163]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7020a03c5b24ef14 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7020a03c5b24ef14"}
16:35:05 [16:35:05.164]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxNjybD/1 with map size 1073741824
16:35:05 [16:35:05.174]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.194]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7020a03c5b24ef14 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7020a03c5b24ef14"}
16:35:05 [16:35:05.195]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7020a03c5b24ef14 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7020a03c5b24ef14"}
16:35:05 [16:35:05.204]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.205]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f63865e4362b10f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f63865e4362b10f"}
16:35:05 [16:35:05.206]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4DU2IK/1 with map size 1073741824
16:35:05 [16:35:05.209]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.230]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1f63865e4362b10f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1f63865e4362b10f"}
16:35:05 [16:35:05.230]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1f63865e4362b10f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1f63865e4362b10f"}
16:35:05 [16:35:05.239]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.239]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40d59b9e06d20f70 epochNumber=0
{"provingJobId":"1:BASE_PARITY:40d59b9e06d20f70"}
16:35:05 [16:35:05.240]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testMdGZ3m/0 with map size 1073741824
16:35:05 [16:35:05.243]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.261]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:40d59b9e06d20f70 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:40d59b9e06d20f70"}
16:35:05 [16:35:05.262]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e303170a71083ae2 epochNumber=0
{"provingJobId":"1:BASE_PARITY:e303170a71083ae2"}
16:35:05 [16:35:05.262]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:40d59b9e06d20f70 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:40d59b9e06d20f70"}
16:35:05 [16:35:05.263]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2adbdee68d5d1094 epochNumber=0
{"provingJobId":"1:BASE_PARITY:2adbdee68d5d1094"}
16:35:05 [16:35:05.281]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.284]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf926b32f46379d5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:bf926b32f46379d5"}
16:35:05 [16:35:05.286]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQ4AkN9/2 with map size 1073741824
16:35:05 [16:35:05.290]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.314]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7890101cc4377ae epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7890101cc4377ae"}
16:35:05 [16:35:05.315]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQ4AkN9/1 with map size 1073741824
16:35:05 [16:35:05.320]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.347]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.348]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e48441cd10ea296 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e48441cd10ea296"}
16:35:05 [16:35:05.349]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH0yQJY/1 with map size 1073741824
16:35:05 [16:35:05.352]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.373]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.374]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b13cc485f20a4bc1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b13cc485f20a4bc1"}
16:35:05 [16:35:05.376]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJefIgJ/1 with map size 1073741824
16:35:05 [16:35:05.379]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.393]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa53c183c229ae20 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa53c183c229ae20"}
16:35:05 [16:35:05.401]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c3c3706e8b0e3d6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1c3c3706e8b0e3d6"}
16:35:05 [16:35:05.402]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJefIgJ/2 with map size 1073741824
16:35:05 [16:35:05.404]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.422]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3e085b9c6f101ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3e085b9c6f101ea"}
16:35:05 [16:35:05.431]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.432]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea1e113a912f725d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea1e113a912f725d"}
16:35:05 [16:35:05.433]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2noSLB/1 with map size 1073741824
16:35:05 [16:35:05.434]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.454]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2f65134b94be22b epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2f65134b94be22b"}
16:35:05 [16:35:05.460]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87befcd3d4ee9a3b epochNumber=2
{"provingJobId":"1:BASE_PARITY:87befcd3d4ee9a3b"}
16:35:05 [16:35:05.461]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2noSLB/2 with map size 1073741824
16:35:05 [16:35:05.463]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.479]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f8f2cba5d324171 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f8f2cba5d324171"}
16:35:05 [16:35:05.488]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.489]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b33e25c6bb172448 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b33e25c6bb172448"}
16:35:05 [16:35:05.490]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBOlr4b/1 with map size 1073741824
16:35:05 [16:35:05.492]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.507]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51d9019661c8f2b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:51d9019661c8f2b3"}
16:35:05 [16:35:05.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a673fb2e1fec69c4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a673fb2e1fec69c4"}
16:35:05 [16:35:05.513]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testBOlr4b/2 with map size 1073741824
16:35:05 [16:35:05.515]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.531]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2650429357faa0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c2650429357faa0e"}
16:35:05 [16:35:05.541]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.542]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc47f805c164c0c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc47f805c164c0c5"}
16:35:05 [16:35:05.543]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFej4Bu/1 with map size 1073741824
16:35:05 [16:35:05.547]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.564]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c87a1b8e99a98d52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c87a1b8e99a98d52"}
16:35:05 [16:35:05.569]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cc47f805c164c0c5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cc47f805c164c0c5"}
16:35:05 [16:35:05.579]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.580]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e10df43f20f3c23b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e10df43f20f3c23b"}
16:35:05 [16:35:05.581]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXcgj83/1 with map size 1073741824
16:35:05 [16:35:05.582]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.598]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:65261cda07608bbc epochNumber=1
{"provingJobId":"1:BASE_PARITY:65261cda07608bbc"}
16:35:05 [16:35:05.603]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e10df43f20f3c23b type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:e10df43f20f3c23b"}
16:35:05 [16:35:05.612]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.612]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2c894ad168547a3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d2c894ad168547a3"}
16:35:05 [16:35:05.613]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testa2cd2i/1 with map size 1073741824
16:35:05 [16:35:05.615]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.632]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:adc93406c2792e42 epochNumber=1
{"provingJobId":"1:BASE_PARITY:adc93406c2792e42"}
16:35:05 [16:35:05.639]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d2c894ad168547a3 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:d2c894ad168547a3"}
16:35:05 [16:35:05.645]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.646]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea4314c39bec6b64 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea4314c39bec6b64"}
16:35:05 [16:35:05.646]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxLIpCG/1 with map size 1073741824
16:35:05 [16:35:05.649]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.665]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ea4314c39bec6b64
{"provingJobId":"1:BASE_PARITY:ea4314c39bec6b64"}
16:35:05 [16:35:05.665]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ea4314c39bec6b64 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ea4314c39bec6b64"}
16:35:05 [16:35:05.672]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b2c8d0db14c20f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b2c8d0db14c20f0"}
16:35:05 [16:35:05.677]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ea4314c39bec6b64 has already been completed
{"provingJobId":"1:BASE_PARITY:ea4314c39bec6b64"}
16:35:05 [16:35:05.680]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.681]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6031e4485af4bcf epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6031e4485af4bcf"}
16:35:05 [16:35:05.681]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZhbhwu/1 with map size 1073741824
16:35:05 [16:35:05.683]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:05 [16:35:05.699]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:986e8fe49f9e75f0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:986e8fe49f9e75f0"}
16:35:05 [16:35:05.700]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZhbhwu/2 with map size 1073741824
16:35:05 [16:35:05.702]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:06 [16:35:06.769]
INFO:
prover-client:proving-broker Proving Broker started
16:35:06 [16:35:06.775]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a6031e4485af4bcf settled=false
{"provingJobId":"1:BASE_PARITY:a6031e4485af4bcf","status":"pending"}
16:35:06 [16:35:06.777]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:986e8fe49f9e75f0 settled=false
{"provingJobId":"1:BASE_PARITY:986e8fe49f9e75f0","status":"pending"}
16:35:06 [16:35:06.778]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a6031e4485af4bcf type=BASE_PARITY startedAt=1750782905717 older agent has taken job
{"provingJobId":"1:BASE_PARITY:a6031e4485af4bcf"}
16:35:06 [16:35:06.779]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a6031e4485af4bcf type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:a6031e4485af4bcf"}
16:35:06 [16:35:06.783]
INFO:
prover-client:proving-broker Proving Broker started
16:35:06 [16:35:06.784]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:819d86006b2fc6d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:819d86006b2fc6d4"}
16:35:06 [16:35:06.785]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDSfZf3/1 with map size 1073741824
16:35:06 [16:35:06.787]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:06 [16:35:06.805]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f0c94b0e05d341a epochNumber=2
{"provingJobId":"1:BASE_PARITY:3f0c94b0e05d341a"}
16:35:06 [16:35:06.806]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDSfZf3/2 with map size 1073741824
16:35:06 [16:35:06.808]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:07 [16:35:07.827]
INFO:
prover-client:proving-broker Proving Broker started
16:35:07 [16:35:07.829]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:819d86006b2fc6d4 settled=false
{"provingJobId":"1:BASE_PARITY:819d86006b2fc6d4","status":"pending"}
16:35:07 [16:35:07.831]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3f0c94b0e05d341a settled=false
{"provingJobId":"1:BASE_PARITY:3f0c94b0e05d341a","status":"pending"}
16:35:07 [16:35:07.832]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:819d86006b2fc6d4 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:819d86006b2fc6d4"}
16:35:07 [16:35:07.841]
INFO:
prover-client:proving-broker Proving Broker started
16:35:07 [16:35:07.842]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:918715d1a6b45593 epochNumber=1
{"provingJobId":"1:BASE_PARITY:918715d1a6b45593"}
16:35:07 [16:35:07.843]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgXcQrs/1 with map size 1073741824
16:35:07 [16:35:07.845]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:07 [16:35:07.865]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:065686768b77685a epochNumber=2
{"provingJobId":"1:BASE_PARITY:065686768b77685a"}
16:35:07 [16:35:07.866]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testgXcQrs/2 with map size 1073741824
16:35:07 [16:35:07.869]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:08 [16:35:08.887]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.890]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:918715d1a6b45593 settled=false
{"provingJobId":"1:BASE_PARITY:918715d1a6b45593","status":"pending"}
16:35:08 [16:35:08.893]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:065686768b77685a settled=false
{"provingJobId":"1:BASE_PARITY:065686768b77685a","status":"pending"}
16:35:08 [16:35:08.894]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:918715d1a6b45593 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:918715d1a6b45593"}
16:35:08 [16:35:08.895]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:918715d1a6b45593 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:918715d1a6b45593"}
16:35:08 [16:35:08.905]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.906]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f4a34e1fc0b6113 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f4a34e1fc0b6113"}
16:35:08 [16:35:08.908]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvqkNRZ/1 with map size 1073741824
16:35:08 [16:35:08.912]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:08 [16:35:08.930]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1b44f71bfea7a73 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e1b44f71bfea7a73"}
16:35:08 [16:35:08.932]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvqkNRZ/2 with map size 1073741824
16:35:08 [16:35:08.935]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:08 [16:35:08.959]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0f4a34e1fc0b6113 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0f4a34e1fc0b6113"}
16:35:08 [16:35:08.966]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e1b44f71bfea7a73 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e1b44f71bfea7a73"}
16:35:08 [16:35:08.978]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.979]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:842a048f9d813854 epochNumber=1
{"provingJobId":"1:BASE_PARITY:842a048f9d813854"}
16:35:08 [16:35:08.980]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4PRmsj/1 with map size 1073741824
16:35:08 [16:35:08.982]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:08 [16:35:09.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17bd5f0578c7e01c epochNumber=2
{"provingJobId":"1:BASE_PARITY:17bd5f0578c7e01c"}
16:35:09 [16:35:09.001]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4PRmsj/2 with map size 1073741824
16:35:09 [16:35:09.005]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.023]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:842a048f9d813854 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:842a048f9d813854"}
16:35:09 [16:35:09.023]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:842a048f9d813854 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:842a048f9d813854"}
16:35:09 [16:35:09.030]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:17bd5f0578c7e01c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:17bd5f0578c7e01c"}
16:35:09 [16:35:09.048]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.049]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a0962e1c6e7086d7 err=test error
{"provingJoId":"1:BASE_PARITY:a0962e1c6e7086d7"}
16:35:09 [16:35:09.052]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.053]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7252c8187340ad3b not found
{"provingJobId":"1:BASE_PARITY:7252c8187340ad3b"}
16:35:09 [16:35:09.056]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.057]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:266869e6fd739937 epochNumber=1
{"provingJobId":"1:BASE_PARITY:266869e6fd739937"}
16:35:09 [16:35:09.058]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testejHU8r/1 with map size 1073741824
16:35:09 [16:35:09.060]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.083]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.083]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:435c202c2a771554 epochNumber=1
{"provingJobId":"1:BASE_PARITY:435c202c2a771554"}
16:35:09 [16:35:09.084]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrAJ7s8/1 with map size 1073741824
16:35:09 [16:35:09.086]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.210]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:435c202c2a771554 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:435c202c2a771554"}
16:35:09 [16:35:09.221]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:023bb3b765ad5468 epochNumber=1
{"provingJobId":"1:BASE_PARITY:023bb3b765ad5468"}
16:35:09 [16:35:09.223]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbAGLKj/1 with map size 1073741824
16:35:09 [16:35:09.225]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.373]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:023bb3b765ad5468 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:023bb3b765ad5468"}
16:35:09 [16:35:09.377]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0520e28cb73b0f85 epochNumber=10
{"provingJobId":"1:BASE_PARITY:0520e28cb73b0f85"}
16:35:09 [16:35:09.378]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testbAGLKj/10 with map size 1073741824
16:35:09 [16:35:09.381]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.424]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:09 [16:35:09.425]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:35:09 [16:35:09.529]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.530]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3fe7186c5643c920 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3fe7186c5643c920"}
16:35:09 [16:35:09.531]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIS44Dl/1 with map size 1073741824
16:35:09 [16:35:09.534]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.707]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3fe7186c5643c920 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:3fe7186c5643c920"}
16:35:09 [16:35:09.761]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0ff2f602775bb6e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c0ff2f602775bb6e"}
16:35:09 [16:35:09.763]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2yKkrh/1 with map size 1073741824
16:35:09 [16:35:09.765]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.784]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c0ff2f602775bb6e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c0ff2f602775bb6e"}
16:35:09 [16:35:09.791]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.792]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0cc8e0279a9bb61 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0cc8e0279a9bb61"}
16:35:09 [16:35:09.798]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testifb6fo/1 with map size 1073741824
16:35:09 [16:35:09.801]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.819]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e0cc8e0279a9bb61 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:e0cc8e0279a9bb61"}
16:35:09 [16:35:09.821]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e0cc8e0279a9bb61 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:e0cc8e0279a9bb61"}
16:35:09 [16:35:09.830]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.831]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3a24da0ef86a856 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3a24da0ef86a856"}
16:35:09 [16:35:09.832]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWkZQrB/1 with map size 1073741824
16:35:09 [16:35:09.834]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.853]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d3a24da0ef86a856 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d3a24da0ef86a856"}
16:35:09 [16:35:09.864]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.865]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64b30e688f477229 epochNumber=1
{"provingJobId":"1:BASE_PARITY:64b30e688f477229"}
16:35:09 [16:35:09.866]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOHXyyM/1 with map size 1073741824
16:35:09 [16:35:09.868]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.885]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:64b30e688f477229 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:64b30e688f477229"}
16:35:09 [16:35:09.886]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a434d7285e7b6557 epochNumber=3
{"provingJobId":"1:BASE_PARITY:a434d7285e7b6557"}
16:35:09 [16:35:09.887]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testOHXyyM/3 with map size 1073741824
16:35:09 [16:35:09.889]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.918]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:09 [16:35:09.918]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:35:09 [16:35:09.933]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:64b30e688f477229 err=test error
{"provingJoId":"1:BASE_PARITY:64b30e688f477229"}
16:35:09 [16:35:09.945]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsVGCfg/1 with map size 1073741824
16:35:09 [16:35:09.955]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.971]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testsVGCfg/2 with map size 1073741824
16:35:09 [16:35:09.976]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.993]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.995]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:99929a9892c08a72 settled=false
{"provingJobId":"1:BASE_PARITY:99929a9892c08a72","status":"pending"}
16:35:09 [16:35:09.997]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:de5f842e8a977bbf settled=false
{"provingJobId":"1:BASE_PARITY:de5f842e8a977bbf","status":"pending"}
16:35:10 [16:35:10.004]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZWYhLv/1 with map size 1073741824
16:35:10 [16:35:10.007]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.022]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZWYhLv/2 with map size 1073741824
16:35:10 [16:35:10.024]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.065]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.067]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:61d3d539cb9674c0 settled=true
{"provingJobId":"1:BASE_PARITY:61d3d539cb9674c0","status":"fulfilled"}
16:35:10 [16:35:10.069]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:eadfec21105157ac settled=true
{"provingJobId":"2:BASE_PARITY:eadfec21105157ac","status":"fulfilled"}
16:35:10 [16:35:10.076]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpTwZPB/1 with map size 1073741824
16:35:10 [16:35:10.079]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.100]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpTwZPB/2 with map size 1073741824
16:35:10 [16:35:10.102]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.118]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.120]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fdd9045105211954 settled=true
{"provingJobId":"1:BASE_PARITY:fdd9045105211954","status":"fulfilled"}
16:35:10 [16:35:10.122]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b41d05270f32ac3e settled=false
{"provingJobId":"1:BASE_PARITY:b41d05270f32ac3e","status":"pending"}
16:35:10 [16:35:10.128]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.129]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:850f83f163e55b13 epochNumber=1
{"provingJobId":"1:BASE_PARITY:850f83f163e55b13"}
16:35:10 [16:35:10.130]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDDiAKO/1 with map size 1073741824
16:35:10 [16:35:10.134]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.164]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.165]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aedc75c1d61fc2b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aedc75c1d61fc2b8"}
16:35:10 [16:35:10.166]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:aedc75c1d61fc2b8: Error: db error: Error: db error
16:35:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:aedc75c1d61fc2b8"}
16:35:10 [16:35:10.169]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.169]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:801beaeb1ac8cd55 epochNumber=1
{"provingJobId":"1:BASE_PARITY:801beaeb1ac8cd55"}
16:35:10 [16:35:10.170]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testovByk9/1 with map size 1073741824
16:35:10 [16:35:10.174]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.193]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:801beaeb1ac8cd55 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:801beaeb1ac8cd55"}
16:35:10 [16:35:10.193]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:801beaeb1ac8cd55 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:801beaeb1ac8cd55"}
16:35:10 [16:35:10.206]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.206]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a53b5443969b0e05 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a53b5443969b0e05"}
16:35:10 [16:35:10.209]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4D3hXp/1 with map size 1073741824
16:35:10 [16:35:10.211]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.231]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a53b5443969b0e05 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a53b5443969b0e05"}
16:35:10 [16:35:10.231]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a53b5443969b0e05 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a53b5443969b0e05"}
16:35:10 [16:35:10.232]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:a53b5443969b0e05: Error: db error
16:35:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:a53b5443969b0e05"}
16:35:10 [16:35:10.236]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d22b022bd86b50fc epochNumber=1
{"provingJobId":"1:BASE_PARITY:d22b022bd86b50fc"}
16:35:10 [16:35:10.237]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDDcoiG/1 with map size 1073741824
16:35:10 [16:35:10.249]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.270]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d22b022bd86b50fc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d22b022bd86b50fc"}
16:35:10 [16:35:10.270]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d22b022bd86b50fc type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d22b022bd86b50fc"}
16:35:10 [16:35:10.280]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f12a26cd645cb0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f12a26cd645cb0b"}
16:35:10 [16:35:10.281]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test94OtZQ/1 with map size 1073741824
16:35:10 [16:35:10.284]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.303]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4f12a26cd645cb0b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4f12a26cd645cb0b"}
16:35:10 [16:35:10.304]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4f12a26cd645cb0b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4f12a26cd645cb0b"}
16:35:10 [16:35:10.304]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:4f12a26cd645cb0b jobErr=test error: Error: db error
16:35:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:4f12a26cd645cb0b"}
16:35:10 [16:35:10.311]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.311]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d0727d859f238320 not found
{"provingJobId":"1:BASE_PARITY:d0727d859f238320"}
16:35:10 [16:35:10.314]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.315]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d588c95679b23516 err=test error
{"provingJoId":"1:BASE_PARITY:d588c95679b23516"}
16:35:10 [16:35:10.319]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.330]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0835de9a8feeb8f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0835de9a8feeb8f0"}
16:35:10 [16:35:10.333]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testepZD7i/1 with map size 1073741824
16:35:10 [16:35:10.341]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.363]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0835de9a8feeb8f0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0835de9a8feeb8f0"}
16:35:10 [16:35:10.364]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0835de9a8feeb8f0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0835de9a8feeb8f0"}
16:35:10 [16:35:10.379]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:2ac878c67f2d971f epochNumber=2
{"provingJobId":"2:BASE_PARITY:2ac878c67f2d971f"}
16:35:10 [16:35:10.380]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testepZD7i/2 with map size 1073741824
16:35:10 [16:35:10.383]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.404]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:2ac878c67f2d971f type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:2ac878c67f2d971f"}
16:35:10 [16:35:10.405]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2ac878c67f2d971f type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:2ac878c67f2d971f"}
16:35:10 [16:35:10.472]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:60c32021daf5d982 epochNumber=3
{"provingJobId":"3:BASE_PARITY:60c32021daf5d982"}
16:35:10 [16:35:10.475]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testepZD7i/3 with map size 1073741824
16:35:10 [16:35:10.478]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.521]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:10 [16:35:10.521]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:35:10 [16:35:10.561]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:c5bbf27a085c1b7c epochNumber=4
{"provingJobId":"4:BASE_PARITY:c5bbf27a085c1b7c"}
16:35:10 [16:35:10.562]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testepZD7i/4 with map size 1073741824
16:35:10 [16:35:10.567]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.601]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:10 [16:35:10.601]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
16:35:10 [16:35:10.656]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:acb3964dff73bc9b epochNumber=5
{"provingJobId":"5:BASE_PARITY:acb3964dff73bc9b"}
16:35:10 [16:35:10.657]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testepZD7i/5 with map size 1073741824
16:35:10 [16:35:10.661]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.708]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:10 [16:35:10.709]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
16:35:10
PASS src/proving_broker/proving_broker.test.ts (
13.373 s)
16:35:10 ProvingBroker
16:35:10 Producer API
16:35:10
✓ refuses stale jobs (52 ms)
16:35:10
✓ enqueues jobs (4 ms)
16:35:10
✓ ignores duplicate jobs (4 ms)
16:35:10
✓ reports correct status when enqueuing repeat jobs (4 ms)
16:35:10
✓ throws an error in case of duplicate job IDs (30 ms)
16:35:10
✓ returns not-found status for non-existing jobs (2 ms)
16:35:10
✓ cancels jobs in queue (3 ms)
16:35:10
✓ cancels jobs in-progress (2 ms)
16:35:10
✓ returns job result if successful (3 ms)
16:35:10
✓ returns job error if failed (3 ms)
16:35:10
✓ correctly returns job status for concurrent writes (3 ms)
16:35:10
✓ refuses stale jobs (42 ms)
16:35:10
✓ enqueues jobs (42 ms)
16:35:10
✓ ignores duplicate jobs (27 ms)
16:35:10
✓ reports correct status when enqueuing repeat jobs (41 ms)
16:35:10
✓ throws an error in case of duplicate job IDs (28 ms)
16:35:10
✓ returns not-found status for non-existing jobs (3 ms)
16:35:10
✓ cancels jobs in queue (32 ms)
16:35:10
✓ cancels jobs in-progress (33 ms)
16:35:10
✓ returns job result if successful (44 ms)
16:35:10
✓ returns job error if failed (35 ms)
16:35:10
✓ correctly returns job status for concurrent writes (42 ms)
16:35:10 Consumer API
16:35:10
✓ returns undefined if no jobs are available (1 ms)
16:35:10
✓ returns jobs in priority order (3 ms)
16:35:10
✓ returns undefined if no jobs are available for the given allowList (2 ms)
16:35:10
✓ returns a job if it is in the allowList (3 ms)
16:35:10
✓ returns the most important job if it is in the allowList (3 ms)
16:35:10
✓ returns any job if filter is empty (2 ms)
16:35:10
✓ returns a new job when reporting job success (3 ms)
16:35:10
✓ returns a new job when reporting permanent error (3 ms)
16:35:10
✓ returns a new job when reporting retry-able error (3 ms)
16:35:10
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
16:35:10
✓ returns a new job if job is already in progress elsewhere (1057 ms)
16:35:10
✓ avoids sending the same job to a new agent after a restart (1004 ms)
16:35:10
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
16:35:10
✓ tracks job result if in progress (3 ms)
16:35:10
✓ tracks job result even if job is in queue (2 ms)
16:35:10
✓ ignores reported job error if unknown job (1 ms)
16:35:10
✓ ignores job result if unknown job (1 ms)
16:35:10
✓ returns undefined if no jobs are available (2 ms)
16:35:10
✓ returns jobs in priority order (61 ms)
16:35:10
✓ returns undefined if no jobs are available for the given allowList (28 ms)
16:35:10
✓ returns a job if it is in the allowList (57 ms)
16:35:10
✓ returns the most important job if it is in the allowList (57 ms)
16:35:10
✓ returns any job if filter is empty (52 ms)
16:35:10
✓ returns a new job when reporting job success (37 ms)
16:35:10
✓ returns a new job when reporting permanent error (34 ms)
16:35:10
✓ returns a new job when reporting retry-able error (32 ms)
16:35:10
✓ returns a new job when reporting progress if current one is cancelled (36 ms)
16:35:10
✓ returns a new job if job is already in progress elsewhere (1102 ms)
16:35:10
✓ avoids sending the same job to a new agent after a restart (1053 ms)
16:35:10
✓ avoids sending a completed job to a new agent after a restart (1068 ms)
16:35:10
✓ tracks job result if in progress (72 ms)
16:35:10
✓ tracks job result even if job is in queue (69 ms)
16:35:10
✓ ignores reported job error if unknown job (5 ms)
16:35:10
✓ ignores job result if unknown job (4 ms)
16:35:10 Timeouts
16:35:10
✓ tracks in progress jobs (2 ms)
16:35:10
✓ re-enqueues jobs that time out (109 ms)
16:35:10
✓ cancel stale jobs that time out (239 ms)
16:35:10
✓ keeps the jobs in progress while it is alive (204 ms)
16:35:10
✓ tracks in progress jobs (26 ms)
16:35:10
✓ re-enqueues jobs that time out (137 ms)
16:35:10
✓ cancel stale jobs that time out (308 ms)
16:35:10
✓ keeps the jobs in progress while it is alive (230 ms)
16:35:10 Retries
16:35:10
✓ retries jobs (3 ms)
16:35:10
✓ retries up to a maximum number of times (2 ms)
16:35:10
✓ passing retry=false does not retry (2 ms)
16:35:10
✓ does not retry if job is stale (27 ms)
16:35:10
✓ retries jobs (27 ms)
16:35:10
✓ retries up to a maximum number of times (41 ms)
16:35:10
✓ passing retry=false does not retry (34 ms)
16:35:10
✓ does not retry if job is stale (73 ms)
16:35:10 Database management
16:35:10
✓ re-enqueues proof requests on start (4 ms)
16:35:10
✓ restores proof results on start (2 ms)
16:35:10
✓ only re-enqueues unfinished jobs (2 ms)
16:35:10
✓ saves job when enqueued (3 ms)
16:35:10
✓ does not retain job if database fails to save (3 ms)
16:35:10
✓ saves job result (2 ms)
16:35:10
✓ saves result even if database fails to save (3 ms)
16:35:10
✓ saves job error (2 ms)
16:35:10
✓ saves job error even if database fails to save (2 ms)
16:35:10
✓ does not save job result if job is unknown (1 ms)
16:35:10
✓ does not save job error if job is unknown (2 ms)
16:35:10
✓ cleans up old jobs periodically (266 ms)
16:35:10
✓ re-enqueues proof requests on start (65 ms)
16:35:10
✓ restores proof results on start (71 ms)
16:35:10
✓ only re-enqueues unfinished jobs (53 ms)
16:35:10
✓ saves job when enqueued (36 ms)
16:35:10
✓ does not retain job if database fails to save (5 ms)
16:35:10
✓ saves job result (36 ms)
16:35:10
✓ saves result even if database fails to save (29 ms)
16:35:10
✓ saves job error (44 ms)
16:35:10
✓ saves job error even if database fails to save (29 ms)
16:35:10
✓ does not save job result if job is unknown (5 ms)
16:35:10
✓ does not save job error if job is unknown (3 ms)
16:35:10
✓ cleans up old jobs periodically (425 ms)
16:35:10
16:35:10
Test Suites: 1 passed, 1 total
16:35:10
Tests: 96 passed, 96 total
16:35:10
Snapshots: 0 total
16:35:10
Time: 13.488 s
16:35:10
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
16:35:10
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?