Parent Log:
http://ci.aztec-labs.com/5c8b0cb269d1f570
Command: a8cc061dbe484831 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/107a625e4a5b124b87c02fa9ff2ae12b5778baa9
Env: REF_NAME=gh-readonly-queue/next/pr-15186-54fc27adbce52a5aa04aeae6eec34ae4b1044f92 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 20 14:24:01 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15186_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
14:24:06 [14:24:06.272]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc6ba431651a739f epochNumber=42
{"provingJobId":"1:BASE_PARITY:dc6ba431651a739f"}
14:24:06 [14:24:06.283]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:0e188da7193277a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e188da7193277a4"}
14:24:06 [14:24:06.321]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.322]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8fe67072047d7254 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8fe67072047d7254"}
14:24:06 [14:24:06.324]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:281137a830c4fa42 epochNumber=1
{"provingJobId":"1:BASE_PARITY:281137a830c4fa42"}
14:24:06 [14:24:06.326]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.326]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce219f78dac24e47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ce219f78dac24e47"}
14:24:06 [14:24:06.327]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ce219f78dac24e47 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ce219f78dac24e47"}
14:24:06 [14:24:06.330]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3fcf9b9132416971 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3fcf9b9132416971"}
14:24:06 [14:24:06.332]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3fcf9b9132416971 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3fcf9b9132416971"}
14:24:06 [14:24:06.332]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3fcf9b9132416971 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3fcf9b9132416971"}
14:24:06 [14:24:06.333]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3fcf9b9132416971 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3fcf9b9132416971"}
14:24:06 [14:24:06.334]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43be1e31393bd55d epochNumber=1
{"provingJobId":"1:BASE_PARITY:43be1e31393bd55d"}
14:24:06 [14:24:06.361]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.362]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.363]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7dadacd8c8ae0585 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7dadacd8c8ae0585"}
14:24:06 [14:24:06.363]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7dadacd8c8ae0585
{"provingJobId":"1:BASE_PARITY:7dadacd8c8ae0585"}
14:24:06 [14:24:06.364]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7dadacd8c8ae0585 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7dadacd8c8ae0585"}
14:24:06 [14:24:06.364]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7dadacd8c8ae0585 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7dadacd8c8ae0585"}
14:24:06 [14:24:06.365]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9dac5d2fbe27659b epochNumber=1
{"provingJobId":"1:BASE_PARITY:9dac5d2fbe27659b"}
14:24:06 [14:24:06.367]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9dac5d2fbe27659b
{"provingJobId":"1:BASE_PARITY:9dac5d2fbe27659b"}
14:24:06 [14:24:06.367]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9dac5d2fbe27659b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:9dac5d2fbe27659b"}
14:24:06 [14:24:06.368]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.369]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd70b51a9b59b08a epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd70b51a9b59b08a"}
14:24:06 [14:24:06.375]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dd70b51a9b59b08a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dd70b51a9b59b08a"}
14:24:06 [14:24:06.376]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dd70b51a9b59b08a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dd70b51a9b59b08a"}
14:24:06 [14:24:06.378]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.378]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:743c4738be5baa8f epochNumber=1
{"provingJobId":"1:BASE_PARITY:743c4738be5baa8f"}
14:24:06 [14:24:06.379]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:743c4738be5baa8f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:743c4738be5baa8f"}
14:24:06 [14:24:06.379]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:743c4738be5baa8f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:743c4738be5baa8f"}
14:24:06 [14:24:06.380]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.381]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53cd89ff09913bee epochNumber=0
{"provingJobId":"1:BASE_PARITY:53cd89ff09913bee"}
14:24:06 [14:24:06.381]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:53cd89ff09913bee epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:53cd89ff09913bee"}
14:24:06 [14:24:06.381]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec2f6de84a9e7785 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ec2f6de84a9e7785"}
14:24:06 [14:24:06.382]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:53cd89ff09913bee epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:53cd89ff09913bee"}
14:24:06 [14:24:06.382]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5868c2120527562e epochNumber=0
{"provingJobId":"1:BASE_PARITY:5868c2120527562e"}
14:24:06 [14:24:06.383]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.385]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.385]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03ea2b6681f955e3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:03ea2b6681f955e3"}
14:24:06 [14:24:06.385]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba3687f5e5894267 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba3687f5e5894267"}
14:24:06 [14:24:06.387]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.387]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a15cc27f1bfa6b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9a15cc27f1bfa6b0"}
14:24:06 [14:24:06.388]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.389]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c274a1a52344f00f epochNumber=1
{"provingJobId":"1:BASE_PARITY:c274a1a52344f00f"}
14:24:06 [14:24:06.389]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:127e55524134a5e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:127e55524134a5e3"}
14:24:06 [14:24:06.390]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f5958330aa78ba6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7f5958330aa78ba6"}
14:24:06 [14:24:06.390]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09b5378eb9048322 epochNumber=1
{"provingJobId":"1:BASE_PARITY:09b5378eb9048322"}
14:24:06 [14:24:06.391]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.392]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ab54fbd9e8a88b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ab54fbd9e8a88b6"}
14:24:06 [14:24:06.392]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bbbf5a8904a45176 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bbbf5a8904a45176"}
14:24:06 [14:24:06.392]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8eb5a85ed5bba9d2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8eb5a85ed5bba9d2"}
14:24:06 [14:24:06.393]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b3f58712611607c epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b3f58712611607c"}
14:24:06 [14:24:06.394]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.395]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eab0f17e22ffa503 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eab0f17e22ffa503"}
14:24:06 [14:24:06.395]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:989ba2069a728273 epochNumber=1
{"provingJobId":"1:BASE_PARITY:989ba2069a728273"}
14:24:06 [14:24:06.395]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a40715ac6483fbb5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a40715ac6483fbb5"}
14:24:06 [14:24:06.396]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82cbf3e9a9f9bd13 epochNumber=1
{"provingJobId":"1:BASE_PARITY:82cbf3e9a9f9bd13"}
14:24:06 [14:24:06.397]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.398]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb2f17ab1ca11aa0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb2f17ab1ca11aa0"}
14:24:06 [14:24:06.398]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:563af6de082bcf92 epochNumber=1
{"provingJobId":"1:BASE_PARITY:563af6de082bcf92"}
14:24:06 [14:24:06.399]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fb2f17ab1ca11aa0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fb2f17ab1ca11aa0"}
14:24:06 [14:24:06.400]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.401]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa022584760cc50a epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa022584760cc50a"}
14:24:06 [14:24:06.401]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b10cf09904ae765 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b10cf09904ae765"}
14:24:06 [14:24:06.402]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fa022584760cc50a type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:fa022584760cc50a"}
14:24:06 [14:24:06.403]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.403]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d826f3df71628f51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d826f3df71628f51"}
14:24:06 [14:24:06.404]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1add2fef58de06a epochNumber=1
{"provingJobId":"1:BASE_PARITY:c1add2fef58de06a"}
14:24:06 [14:24:06.404]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d826f3df71628f51 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:d826f3df71628f51"}
14:24:06 [14:24:06.406]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.406]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60065273bb055562 epochNumber=1
{"provingJobId":"1:BASE_PARITY:60065273bb055562"}
14:24:06 [14:24:06.407]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:60065273bb055562
{"provingJobId":"1:BASE_PARITY:60065273bb055562"}
14:24:06 [14:24:06.407]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:60065273bb055562 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:60065273bb055562"}
14:24:06 [14:24:06.407]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bfd95a43eab22514 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bfd95a43eab22514"}
14:24:06 [14:24:06.408]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:60065273bb055562 has already been completed
{"provingJobId":"1:BASE_PARITY:60065273bb055562"}
14:24:06 [14:24:06.409]
INFO:
prover-client:proving-broker Proving Broker started
14:24:06 [14:24:06.409]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0a276d739092243 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0a276d739092243"}
14:24:06 [14:24:06.410]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:118be06c01665b00 epochNumber=2
{"provingJobId":"1:BASE_PARITY:118be06c01665b00"}
14:24:07 [14:24:07.461]
INFO:
prover-client:proving-broker Proving Broker started
14:24:07 [14:24:07.462]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b0a276d739092243 settled=false
{"provingJobId":"1:BASE_PARITY:b0a276d739092243","status":"pending"}
14:24:07 [14:24:07.462]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:118be06c01665b00 settled=false
{"provingJobId":"1:BASE_PARITY:118be06c01665b00","status":"pending"}
14:24:07 [14:24:07.463]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b0a276d739092243 type=BASE_PARITY startedAt=1750429446410 older agent has taken job
{"provingJobId":"1:BASE_PARITY:b0a276d739092243"}
14:24:07 [14:24:07.464]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b0a276d739092243 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:b0a276d739092243"}
14:24:07 [14:24:07.465]
INFO:
prover-client:proving-broker Proving Broker started
14:24:07 [14:24:07.466]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:27b8b671d78e93aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:27b8b671d78e93aa"}
14:24:07 [14:24:07.466]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58b6df4137525b0f epochNumber=2
{"provingJobId":"1:BASE_PARITY:58b6df4137525b0f"}
14:24:08 [14:24:08.467]
INFO:
prover-client:proving-broker Proving Broker started
14:24:08 [14:24:08.468]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:27b8b671d78e93aa settled=false
{"provingJobId":"1:BASE_PARITY:27b8b671d78e93aa","status":"pending"}
14:24:08 [14:24:08.468]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:58b6df4137525b0f settled=false
{"provingJobId":"1:BASE_PARITY:58b6df4137525b0f","status":"pending"}
14:24:08 [14:24:08.468]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:27b8b671d78e93aa type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:27b8b671d78e93aa"}
14:24:08 [14:24:08.470]
INFO:
prover-client:proving-broker Proving Broker started
14:24:08 [14:24:08.470]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:687391c20a2b26fc epochNumber=1
{"provingJobId":"1:BASE_PARITY:687391c20a2b26fc"}
14:24:08 [14:24:08.471]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7d5ed9d722559437 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7d5ed9d722559437"}
14:24:09 [14:24:09.471]
INFO:
prover-client:proving-broker Proving Broker started
14:24:09 [14:24:09.472]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:687391c20a2b26fc settled=false
{"provingJobId":"1:BASE_PARITY:687391c20a2b26fc","status":"pending"}
14:24:09 [14:24:09.472]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7d5ed9d722559437 settled=false
{"provingJobId":"1:BASE_PARITY:7d5ed9d722559437","status":"pending"}
14:24:09 [14:24:09.473]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:687391c20a2b26fc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:687391c20a2b26fc"}
14:24:09 [14:24:09.473]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:687391c20a2b26fc type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:687391c20a2b26fc"}
14:24:09 [14:24:09.475]
INFO:
prover-client:proving-broker Proving Broker started
14:24:09 [14:24:09.476]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a2304475730e4d6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a2304475730e4d6a"}
14:24:09 [14:24:09.476]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5cf071a0a0d6dca epochNumber=2
{"provingJobId":"1:BASE_PARITY:e5cf071a0a0d6dca"}
14:24:09 [14:24:09.476]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a2304475730e4d6a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a2304475730e4d6a"}
14:24:09 [14:24:09.477]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e5cf071a0a0d6dca type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e5cf071a0a0d6dca"}
14:24:09 [14:24:09.477]
INFO:
prover-client:proving-broker Proving Broker started
14:24:09 [14:24:09.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a8bea109ca41d6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a8bea109ca41d6c"}
14:24:09 [14:24:09.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc419f4957eaed78 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cc419f4957eaed78"}
14:24:09 [14:24:09.478]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8a8bea109ca41d6c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8a8bea109ca41d6c"}
14:24:09 [14:24:09.478]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8a8bea109ca41d6c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8a8bea109ca41d6c"}
14:24:09 [14:24:09.479]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cc419f4957eaed78 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:cc419f4957eaed78"}
14:24:09 [14:24:09.479]
INFO:
prover-client:proving-broker Proving Broker started
14:24:09 [14:24:09.480]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:88517628db95f417 err=test error
{"provingJoId":"1:BASE_PARITY:88517628db95f417"}
14:24:09 [14:24:09.481]
INFO:
prover-client:proving-broker Proving Broker started
14:24:09 [14:24:09.481]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c6b691a68759a216 not found
{"provingJobId":"1:BASE_PARITY:c6b691a68759a216"}
14:24:09 [14:24:09.482]
INFO:
prover-client:proving-broker Proving Broker started
14:24:09 [14:24:09.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc9e9f6068fd20ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc9e9f6068fd20ca"}
14:24:09 [14:24:09.484]
INFO:
prover-client:proving-broker Proving Broker started
14:24:09 [14:24:09.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a460eccdfeb91909 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a460eccdfeb91909"}
14:24:09 [14:24:09.586]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a460eccdfeb91909 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a460eccdfeb91909"}
14:24:09 [14:24:09.594]
INFO:
prover-client:proving-broker Proving Broker started
14:24:09 [14:24:09.595]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60ce6cb0b8764607 epochNumber=1
{"provingJobId":"1:BASE_PARITY:60ce6cb0b8764607"}
14:24:09 [14:24:09.697]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:60ce6cb0b8764607 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:60ce6cb0b8764607"}
14:24:09 [14:24:09.704]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:410e770a8fd813f9 epochNumber=10
{"provingJobId":"1:BASE_PARITY:410e770a8fd813f9"}
14:24:09 [14:24:09.723]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:09 [14:24:09.831]
INFO:
prover-client:proving-broker Proving Broker started
14:24:09 [14:24:09.831]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f1ae8e96874ef7fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:f1ae8e96874ef7fb"}
14:24:09 [14:24:09.984]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f1ae8e96874ef7fb timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f1ae8e96874ef7fb"}
14:24:10 [14:24:10.034]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.035]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32493ecf4f5a74db epochNumber=1
{"provingJobId":"1:BASE_PARITY:32493ecf4f5a74db"}
14:24:10 [14:24:10.036]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:32493ecf4f5a74db type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:32493ecf4f5a74db"}
14:24:10 [14:24:10.037]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.037]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:669e9f7311cb5492 epochNumber=1
{"provingJobId":"1:BASE_PARITY:669e9f7311cb5492"}
14:24:10 [14:24:10.038]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:669e9f7311cb5492 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:669e9f7311cb5492"}
14:24:10 [14:24:10.038]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:669e9f7311cb5492 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:669e9f7311cb5492"}
14:24:10 [14:24:10.039]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.040]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a60706bcc3d742e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a60706bcc3d742e8"}
14:24:10 [14:24:10.040]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a60706bcc3d742e8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a60706bcc3d742e8"}
14:24:10 [14:24:10.041]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.041]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3504a5538a32e99 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3504a5538a32e99"}
14:24:10 [14:24:10.041]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d3504a5538a32e99 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d3504a5538a32e99"}
14:24:10 [14:24:10.042]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f232a0827446e48 epochNumber=3
{"provingJobId":"1:BASE_PARITY:2f232a0827446e48"}
14:24:10 [14:24:10.066]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:10 [14:24:10.068]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d3504a5538a32e99 err=test error
{"provingJoId":"1:BASE_PARITY:d3504a5538a32e99"}
14:24:10 [14:24:10.070]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.070]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:632d4faf9fd79099 settled=false
{"provingJobId":"1:BASE_PARITY:632d4faf9fd79099","status":"pending"}
14:24:10 [14:24:10.070]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4d9da84e03eebcfb settled=false
{"provingJobId":"1:BASE_PARITY:4d9da84e03eebcfb","status":"pending"}
14:24:10 [14:24:10.073]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.073]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:86e38ea9766e81e2 settled=true
{"provingJobId":"1:BASE_PARITY:86e38ea9766e81e2","status":"fulfilled"}
14:24:10 [14:24:10.073]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:6ac06fba59b7d1eb settled=true
{"provingJobId":"2:BASE_PARITY:6ac06fba59b7d1eb","status":"fulfilled"}
14:24:10 [14:24:10.074]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.075]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:23df48ae8ec68259 settled=true
{"provingJobId":"1:BASE_PARITY:23df48ae8ec68259","status":"fulfilled"}
14:24:10 [14:24:10.075]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8699d03e65415f1e settled=false
{"provingJobId":"1:BASE_PARITY:8699d03e65415f1e","status":"pending"}
14:24:10 [14:24:10.076]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f3e01d16dded289 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f3e01d16dded289"}
14:24:10 [14:24:10.079]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d73391f6f3b9a8ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:d73391f6f3b9a8ca"}
14:24:10 [14:24:10.080]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:d73391f6f3b9a8ca: Error: db error: Error: db error
14:24:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:d73391f6f3b9a8ca"}
14:24:10 [14:24:10.081]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9685126bb769e255 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9685126bb769e255"}
14:24:10 [14:24:10.082]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9685126bb769e255 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9685126bb769e255"}
14:24:10 [14:24:10.082]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9685126bb769e255 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9685126bb769e255"}
14:24:10 [14:24:10.083]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.084]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f93ea7c24a5eb140 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f93ea7c24a5eb140"}
14:24:10 [14:24:10.084]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f93ea7c24a5eb140 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f93ea7c24a5eb140"}
14:24:10 [14:24:10.084]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f93ea7c24a5eb140 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f93ea7c24a5eb140"}
14:24:10 [14:24:10.085]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:f93ea7c24a5eb140: Error: db error
14:24:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:f93ea7c24a5eb140"}
14:24:10 [14:24:10.086]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a63e4b66ff4508d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a63e4b66ff4508d9"}
14:24:10 [14:24:10.087]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a63e4b66ff4508d9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a63e4b66ff4508d9"}
14:24:10 [14:24:10.087]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a63e4b66ff4508d9 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a63e4b66ff4508d9"}
14:24:10 [14:24:10.088]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.089]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:544dfb760a1702e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:544dfb760a1702e2"}
14:24:10 [14:24:10.089]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:544dfb760a1702e2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:544dfb760a1702e2"}
14:24:10 [14:24:10.089]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:544dfb760a1702e2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:544dfb760a1702e2"}
14:24:10 [14:24:10.090]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:544dfb760a1702e2 jobErr=test error: Error: db error
14:24:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:544dfb760a1702e2"}
14:24:10 [14:24:10.091]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.091]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:61573bd0be5cc0f6 not found
{"provingJobId":"1:BASE_PARITY:61573bd0be5cc0f6"}
14:24:10 [14:24:10.092]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.093]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ac609de291574166 err=test error
{"provingJoId":"1:BASE_PARITY:ac609de291574166"}
14:24:10 [14:24:10.094]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.104]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed61ba92e3d53d4a epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed61ba92e3d53d4a"}
14:24:10 [14:24:10.104]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ed61ba92e3d53d4a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ed61ba92e3d53d4a"}
14:24:10 [14:24:10.104]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ed61ba92e3d53d4a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ed61ba92e3d53d4a"}
14:24:10 [14:24:10.114]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:bbf8d43ee9e4cdd6 epochNumber=2
{"provingJobId":"2:BASE_PARITY:bbf8d43ee9e4cdd6"}
14:24:10 [14:24:10.114]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:bbf8d43ee9e4cdd6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:bbf8d43ee9e4cdd6"}
14:24:10 [14:24:10.115]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:bbf8d43ee9e4cdd6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:bbf8d43ee9e4cdd6"}
14:24:10 [14:24:10.177]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:75aa31cd400f3e05 epochNumber=3
{"provingJobId":"3:BASE_PARITY:75aa31cd400f3e05"}
14:24:10 [14:24:10.196]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:10 [14:24:10.238]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:6e3dc4a820560b79 epochNumber=4
{"provingJobId":"4:BASE_PARITY:6e3dc4a820560b79"}
14:24:10 [14:24:10.247]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:10 [14:24:10.303]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:9ab7ab83f4871005 epochNumber=5
{"provingJobId":"5:BASE_PARITY:9ab7ab83f4871005"}
14:24:10 [14:24:10.323]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:10 [14:24:10.357]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.358]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd9841d29969aa96 epochNumber=42
{"provingJobId":"1:BASE_PARITY:dd9841d29969aa96"}
14:24:10 [14:24:10.359]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testASTIhI/42 with map size 1073741824
14:24:10 [14:24:10.363]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.387]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:7916f79eaef78640 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7916f79eaef78640"}
14:24:10 [14:24:10.395]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.395]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ad2442dfc61ad744 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ad2442dfc61ad744"}
14:24:10 [14:24:10.397]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFVyu18/1 with map size 1073741824
14:24:10 [14:24:10.399]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.420]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b4e70da8cdd78c4a epochNumber=1
{"provingJobId":"1:BASE_PARITY:b4e70da8cdd78c4a"}
14:24:10 [14:24:10.429]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.430]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:27b32eff426ad938 epochNumber=1
{"provingJobId":"1:BASE_PARITY:27b32eff426ad938"}
14:24:10 [14:24:10.431]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testv6SAi8/1 with map size 1073741824
14:24:10 [14:24:10.434]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.456]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:27b32eff426ad938 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:27b32eff426ad938"}
14:24:10 [14:24:10.460]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.461]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ff76e54e2bc2874 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ff76e54e2bc2874"}
14:24:10 [14:24:10.462]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBBJafk/1 with map size 1073741824
14:24:10 [14:24:10.465]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.483]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7ff76e54e2bc2874 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7ff76e54e2bc2874"}
14:24:10 [14:24:10.483]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7ff76e54e2bc2874 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7ff76e54e2bc2874"}
14:24:10 [14:24:10.489]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7ff76e54e2bc2874 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7ff76e54e2bc2874"}
14:24:10 [14:24:10.499]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd97d495e110b789 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd97d495e110b789"}
14:24:10 [14:24:10.500]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyT3blP/1 with map size 1073741824
14:24:10 [14:24:10.514]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.537]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.540]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d11e599a76906300 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d11e599a76906300"}
14:24:10 [14:24:10.542]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDuq3im/1 with map size 1073741824
14:24:10 [14:24:10.544]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.561]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d11e599a76906300
{"provingJobId":"1:BASE_PARITY:d11e599a76906300"}
14:24:10 [14:24:10.562]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d11e599a76906300 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d11e599a76906300"}
14:24:10 [14:24:10.562]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d11e599a76906300 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:d11e599a76906300"}
14:24:10 [14:24:10.571]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.572]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc93e7a0e03e67a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc93e7a0e03e67a5"}
14:24:10 [14:24:10.572]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRSua05/1 with map size 1073741824
14:24:10 [14:24:10.574]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.595]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:fc93e7a0e03e67a5
{"provingJobId":"1:BASE_PARITY:fc93e7a0e03e67a5"}
14:24:10 [14:24:10.596]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fc93e7a0e03e67a5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:fc93e7a0e03e67a5"}
14:24:10 [14:24:10.604]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.605]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:74cadd1815f4525c epochNumber=1
{"provingJobId":"1:BASE_PARITY:74cadd1815f4525c"}
14:24:10 [14:24:10.610]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaWxSOC/1 with map size 1073741824
14:24:10 [14:24:10.613]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.630]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:74cadd1815f4525c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:74cadd1815f4525c"}
14:24:10 [14:24:10.631]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:74cadd1815f4525c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:74cadd1815f4525c"}
14:24:10 [14:24:10.639]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.641]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dde1ef146e488a4e epochNumber=1
{"provingJobId":"1:BASE_PARITY:dde1ef146e488a4e"}
14:24:10 [14:24:10.642]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5jrq8W/1 with map size 1073741824
14:24:10 [14:24:10.646]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.663]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dde1ef146e488a4e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dde1ef146e488a4e"}
14:24:10 [14:24:10.664]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dde1ef146e488a4e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:dde1ef146e488a4e"}
14:24:10 [14:24:10.672]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.673]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:580b420615663943 epochNumber=0
{"provingJobId":"1:BASE_PARITY:580b420615663943"}
14:24:10 [14:24:10.674]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testYiVfyK/0 with map size 1073741824
14:24:10 [14:24:10.676]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.697]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:580b420615663943 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:580b420615663943"}
14:24:10 [14:24:10.698]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df1bd6a31e1c3e80 epochNumber=0
{"provingJobId":"1:BASE_PARITY:df1bd6a31e1c3e80"}
14:24:10 [14:24:10.698]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:580b420615663943 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:580b420615663943"}
14:24:10 [14:24:10.698]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8f4dd0e3f137265 epochNumber=0
{"provingJobId":"1:BASE_PARITY:b8f4dd0e3f137265"}
14:24:10 [14:24:10.714]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.716]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e285cca8f52b54b epochNumber=2
{"provingJobId":"1:BASE_PARITY:4e285cca8f52b54b"}
14:24:10 [14:24:10.718]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0LPQTX/2 with map size 1073741824
14:24:10 [14:24:10.719]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.734]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ce7eb1a175ecb16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ce7eb1a175ecb16"}
14:24:10 [14:24:10.735]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0LPQTX/1 with map size 1073741824
14:24:10 [14:24:10.737]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.755]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.756]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:745b15b55ae7b177 epochNumber=1
{"provingJobId":"1:BASE_PARITY:745b15b55ae7b177"}
14:24:10 [14:24:10.756]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf7yWQm/1 with map size 1073741824
14:24:10 [14:24:10.758]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.777]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4d55834f683a369 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4d55834f683a369"}
14:24:10 [14:24:10.779]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyYxjXg/1 with map size 1073741824
14:24:10 [14:24:10.781]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.797]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:234937d33dab7f5c epochNumber=1
{"provingJobId":"1:BASE_PARITY:234937d33dab7f5c"}
14:24:10 [14:24:10.803]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:499ec4842bb3b0a0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:499ec4842bb3b0a0"}
14:24:10 [14:24:10.804]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyYxjXg/2 with map size 1073741824
14:24:10 [14:24:10.806]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.822]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83d88fb2cc7b4192 epochNumber=1
{"provingJobId":"1:BASE_PARITY:83d88fb2cc7b4192"}
14:24:10 [14:24:10.830]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.831]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02912b2e8317551c epochNumber=1
{"provingJobId":"1:BASE_PARITY:02912b2e8317551c"}
14:24:10 [14:24:10.831]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5hIojU/1 with map size 1073741824
14:24:10 [14:24:10.833]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.848]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c1882ff032eddd0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c1882ff032eddd0"}
14:24:10 [14:24:10.854]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c5233b7f43111e7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6c5233b7f43111e7"}
14:24:10 [14:24:10.855]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5hIojU/2 with map size 1073741824
14:24:10 [14:24:10.857]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:737075bfa9360b84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:737075bfa9360b84"}
14:24:10 [14:24:10.882]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.882]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:908aedaa13a22b6e epochNumber=1
{"provingJobId":"1:BASE_PARITY:908aedaa13a22b6e"}
14:24:10 [14:24:10.883]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJZKKqi/1 with map size 1073741824
14:24:10 [14:24:10.885]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.900]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d9684922bc71b5f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d9684922bc71b5f5"}
14:24:10 [14:24:10.906]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b25f0021bcb46155 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b25f0021bcb46155"}
14:24:10 [14:24:10.907]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJZKKqi/2 with map size 1073741824
14:24:10 [14:24:10.909]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.925]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3d56750986e09e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3d56750986e09e1"}
14:24:10 [14:24:10.933]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.934]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99c84c9a158ed1ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:99c84c9a158ed1ea"}
14:24:10 [14:24:10.934]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2pboka/1 with map size 1073741824
14:24:10 [14:24:10.939]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.957]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b968ee97a36ff342 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b968ee97a36ff342"}
14:24:10 [14:24:10.963]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:99c84c9a158ed1ea type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:99c84c9a158ed1ea"}
14:24:10 [14:24:10.971]
INFO:
prover-client:proving-broker Proving Broker started
14:24:10 [14:24:10.972]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:551c99f48f2f886c epochNumber=1
{"provingJobId":"1:BASE_PARITY:551c99f48f2f886c"}
14:24:10 [14:24:10.972]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYrRout/1 with map size 1073741824
14:24:10 [14:24:10.974]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:10 [14:24:10.989]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41edac85420d8eaa epochNumber=1
{"provingJobId":"1:BASE_PARITY:41edac85420d8eaa"}
14:24:10 [14:24:10.995]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:551c99f48f2f886c type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:551c99f48f2f886c"}
14:24:11 [14:24:11.003]
INFO:
prover-client:proving-broker Proving Broker started
14:24:11 [14:24:11.003]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bfdb094d00f0e5da epochNumber=1
{"provingJobId":"1:BASE_PARITY:bfdb094d00f0e5da"}
14:24:11 [14:24:11.004]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqcvbrj/1 with map size 1073741824
14:24:11 [14:24:11.005]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:11 [14:24:11.021]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8ce1d6b7385dd0d epochNumber=1
{"provingJobId":"1:BASE_PARITY:e8ce1d6b7385dd0d"}
14:24:11 [14:24:11.026]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bfdb094d00f0e5da type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:bfdb094d00f0e5da"}
14:24:11 [14:24:11.029]
INFO:
prover-client:proving-broker Proving Broker started
14:24:11 [14:24:11.030]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f54bd5204656039 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f54bd5204656039"}
14:24:11 [14:24:11.030]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testV7EET5/1 with map size 1073741824
14:24:11 [14:24:11.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:11 [14:24:11.047]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4f54bd5204656039
{"provingJobId":"1:BASE_PARITY:4f54bd5204656039"}
14:24:11 [14:24:11.047]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4f54bd5204656039 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4f54bd5204656039"}
14:24:11 [14:24:11.053]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c99a49f9204af10 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c99a49f9204af10"}
14:24:11 [14:24:11.058]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4f54bd5204656039 has already been completed
{"provingJobId":"1:BASE_PARITY:4f54bd5204656039"}
14:24:11 [14:24:11.061]
INFO:
prover-client:proving-broker Proving Broker started
14:24:11 [14:24:11.062]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eba049615131824b epochNumber=1
{"provingJobId":"1:BASE_PARITY:eba049615131824b"}
14:24:11 [14:24:11.063]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzrLz2e/1 with map size 1073741824
14:24:11 [14:24:11.067]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:11 [14:24:11.085]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8127d41f6ef9b9ef epochNumber=2
{"provingJobId":"1:BASE_PARITY:8127d41f6ef9b9ef"}
14:24:11 [14:24:11.086]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testzrLz2e/2 with map size 1073741824
14:24:11 [14:24:11.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:12 [14:24:12.158]
INFO:
prover-client:proving-broker Proving Broker started
14:24:12 [14:24:12.164]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:eba049615131824b settled=false
{"provingJobId":"1:BASE_PARITY:eba049615131824b","status":"pending"}
14:24:12 [14:24:12.165]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8127d41f6ef9b9ef settled=false
{"provingJobId":"1:BASE_PARITY:8127d41f6ef9b9ef","status":"pending"}
14:24:12 [14:24:12.166]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eba049615131824b type=BASE_PARITY startedAt=1750429451105 older agent has taken job
{"provingJobId":"1:BASE_PARITY:eba049615131824b"}
14:24:12 [14:24:12.166]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eba049615131824b type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:eba049615131824b"}
14:24:12 [14:24:12.169]
INFO:
prover-client:proving-broker Proving Broker started
14:24:12 [14:24:12.170]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8eaa6bb888747685 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8eaa6bb888747685"}
14:24:12 [14:24:12.170]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPBtpQX/1 with map size 1073741824
14:24:12 [14:24:12.173]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:12 [14:24:12.191]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a3683f40cec5b16 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4a3683f40cec5b16"}
14:24:12 [14:24:12.197]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testPBtpQX/2 with map size 1073741824
14:24:12 [14:24:12.203]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:13 [14:24:13.223]
INFO:
prover-client:proving-broker Proving Broker started
14:24:13 [14:24:13.226]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8eaa6bb888747685 settled=false
{"provingJobId":"1:BASE_PARITY:8eaa6bb888747685","status":"pending"}
14:24:13 [14:24:13.227]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4a3683f40cec5b16 settled=false
{"provingJobId":"1:BASE_PARITY:4a3683f40cec5b16","status":"pending"}
14:24:13 [14:24:13.228]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8eaa6bb888747685 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:8eaa6bb888747685"}
14:24:13 [14:24:13.232]
INFO:
prover-client:proving-broker Proving Broker started
14:24:13 [14:24:13.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a4e5ccb28a9d93a epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a4e5ccb28a9d93a"}
14:24:13 [14:24:13.233]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8n24HL/1 with map size 1073741824
14:24:13 [14:24:13.235]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:13 [14:24:13.257]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d54a588cddbf80ee epochNumber=2
{"provingJobId":"1:BASE_PARITY:d54a588cddbf80ee"}
14:24:13 [14:24:13.258]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8n24HL/2 with map size 1073741824
14:24:13 [14:24:13.265]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:14 [14:24:14.282]
INFO:
prover-client:proving-broker Proving Broker started
14:24:14 [14:24:14.285]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7a4e5ccb28a9d93a settled=false
{"provingJobId":"1:BASE_PARITY:7a4e5ccb28a9d93a","status":"pending"}
14:24:14 [14:24:14.287]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d54a588cddbf80ee settled=false
{"provingJobId":"1:BASE_PARITY:d54a588cddbf80ee","status":"pending"}
14:24:14 [14:24:14.288]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7a4e5ccb28a9d93a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7a4e5ccb28a9d93a"}
14:24:14 [14:24:14.289]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7a4e5ccb28a9d93a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7a4e5ccb28a9d93a"}
14:24:14 [14:24:14.299]
INFO:
prover-client:proving-broker Proving Broker started
14:24:14 [14:24:14.300]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5634f7223e6298d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5634f7223e6298d2"}
14:24:14 [14:24:14.301]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtaXEOV/1 with map size 1073741824
14:24:14 [14:24:14.302]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:14 [14:24:14.318]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0237813c20535e43 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0237813c20535e43"}
14:24:14 [14:24:14.319]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtaXEOV/2 with map size 1073741824
14:24:14 [14:24:14.321]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:14 [14:24:14.339]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5634f7223e6298d2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5634f7223e6298d2"}
14:24:14 [14:24:14.346]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0237813c20535e43 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0237813c20535e43"}
14:24:14 [14:24:14.357]
INFO:
prover-client:proving-broker Proving Broker started
14:24:14 [14:24:14.358]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14deac665694759c epochNumber=1
{"provingJobId":"1:BASE_PARITY:14deac665694759c"}
14:24:14 [14:24:14.359]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test97uetI/1 with map size 1073741824
14:24:14 [14:24:14.361]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:14 [14:24:14.380]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:371ea48456ae0864 epochNumber=2
{"provingJobId":"1:BASE_PARITY:371ea48456ae0864"}
14:24:14 [14:24:14.381]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test97uetI/2 with map size 1073741824
14:24:14 [14:24:14.384]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:14 [14:24:14.402]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:14deac665694759c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:14deac665694759c"}
14:24:14 [14:24:14.403]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:14deac665694759c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:14deac665694759c"}
14:24:14 [14:24:14.409]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:371ea48456ae0864 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:371ea48456ae0864"}
14:24:14 [14:24:14.421]
INFO:
prover-client:proving-broker Proving Broker started
14:24:14 [14:24:14.423]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:9ddbf5d1304306c9 err=test error
{"provingJoId":"1:BASE_PARITY:9ddbf5d1304306c9"}
14:24:14 [14:24:14.426]
INFO:
prover-client:proving-broker Proving Broker started
14:24:14 [14:24:14.427]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1b056427397e40a8 not found
{"provingJobId":"1:BASE_PARITY:1b056427397e40a8"}
14:24:14 [14:24:14.431]
INFO:
prover-client:proving-broker Proving Broker started
14:24:14 [14:24:14.432]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32697dc5fcc1a71f epochNumber=1
{"provingJobId":"1:BASE_PARITY:32697dc5fcc1a71f"}
14:24:14 [14:24:14.433]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZEDZde/1 with map size 1073741824
14:24:14 [14:24:14.435]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:14 [14:24:14.461]
INFO:
prover-client:proving-broker Proving Broker started
14:24:14 [14:24:14.462]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e0f91829e5138bd epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e0f91829e5138bd"}
14:24:14 [14:24:14.463]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testByiIX2/1 with map size 1073741824
14:24:14 [14:24:14.467]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:14 [14:24:14.588]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9e0f91829e5138bd timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9e0f91829e5138bd"}
14:24:14 [14:24:14.596]
INFO:
prover-client:proving-broker Proving Broker started
14:24:14 [14:24:14.597]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c639c60d0f4ef22 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c639c60d0f4ef22"}
14:24:14 [14:24:14.598]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjLOYa0/1 with map size 1073741824
14:24:14 [14:24:14.600]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:14 [14:24:14.724]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6c639c60d0f4ef22 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6c639c60d0f4ef22"}
14:24:14 [14:24:14.729]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e17683205d6a76a7 epochNumber=10
{"provingJobId":"1:BASE_PARITY:e17683205d6a76a7"}
14:24:14 [14:24:14.730]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testjLOYa0/10 with map size 1073741824
14:24:14 [14:24:14.733]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:14 [14:24:14.775]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:14 [14:24:14.776]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:24:14 [14:24:14.885]
INFO:
prover-client:proving-broker Proving Broker started
14:24:14 [14:24:14.886]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58b53b8c63903090 epochNumber=1
{"provingJobId":"1:BASE_PARITY:58b53b8c63903090"}
14:24:14 [14:24:14.887]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoEiHLA/1 with map size 1073741824
14:24:14 [14:24:14.888]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.094]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:58b53b8c63903090 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:58b53b8c63903090"}
14:24:15 [14:24:15.132]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9c0c421a71264b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c9c0c421a71264b1"}
14:24:15 [14:24:15.133]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8WmxIq/1 with map size 1073741824
14:24:15 [14:24:15.135]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.152]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c9c0c421a71264b1 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c9c0c421a71264b1"}
14:24:15 [14:24:15.155]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.156]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e8c80b75bcf1f25 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e8c80b75bcf1f25"}
14:24:15 [14:24:15.156]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAH46RX/1 with map size 1073741824
14:24:15 [14:24:15.158]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.177]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8e8c80b75bcf1f25 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:8e8c80b75bcf1f25"}
14:24:15 [14:24:15.178]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8e8c80b75bcf1f25 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:8e8c80b75bcf1f25"}
14:24:15 [14:24:15.189]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.196]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85434080e5dfc48d epochNumber=1
{"provingJobId":"1:BASE_PARITY:85434080e5dfc48d"}
14:24:15 [14:24:15.207]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPjpDSi/1 with map size 1073741824
14:24:15 [14:24:15.219]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.240]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:85434080e5dfc48d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:85434080e5dfc48d"}
14:24:15 [14:24:15.252]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8775cc3806897955 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8775cc3806897955"}
14:24:15 [14:24:15.254]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testi7IfPU/1 with map size 1073741824
14:24:15 [14:24:15.257]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.279]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8775cc3806897955 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:8775cc3806897955"}
14:24:15 [14:24:15.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a61ff39f4b26d6e9 epochNumber=3
{"provingJobId":"1:BASE_PARITY:a61ff39f4b26d6e9"}
14:24:15 [14:24:15.282]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testi7IfPU/3 with map size 1073741824
14:24:15 [14:24:15.284]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.332]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:15 [14:24:15.333]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:24:15 [14:24:15.344]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8775cc3806897955 err=test error
{"provingJoId":"1:BASE_PARITY:8775cc3806897955"}
14:24:15 [14:24:15.349]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testscw7q4/1 with map size 1073741824
14:24:15 [14:24:15.353]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.372]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testscw7q4/2 with map size 1073741824
14:24:15 [14:24:15.375]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.404]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.412]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:004930829f1462d8 settled=false
{"provingJobId":"1:BASE_PARITY:004930829f1462d8","status":"pending"}
14:24:15 [14:24:15.416]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:672fdab4d0c00035 settled=false
{"provingJobId":"1:BASE_PARITY:672fdab4d0c00035","status":"pending"}
14:24:15 [14:24:15.428]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQTTBeE/1 with map size 1073741824
14:24:15 [14:24:15.432]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.453]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQTTBeE/2 with map size 1073741824
14:24:15 [14:24:15.455]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.486]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.490]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a2377e952f957fe4 settled=true
{"provingJobId":"1:BASE_PARITY:a2377e952f957fe4","status":"fulfilled"}
14:24:15 [14:24:15.498]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:4722d8630f075594 settled=true
{"provingJobId":"2:BASE_PARITY:4722d8630f075594","status":"fulfilled"}
14:24:15 [14:24:15.518]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTF6Kbx/1 with map size 1073741824
14:24:15 [14:24:15.521]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTF6Kbx/2 with map size 1073741824
14:24:15 [14:24:15.551]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.571]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.574]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:43ab9e0b598daade settled=true
{"provingJobId":"1:BASE_PARITY:43ab9e0b598daade","status":"fulfilled"}
14:24:15 [14:24:15.576]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e8bfe1eff2cc5839 settled=false
{"provingJobId":"1:BASE_PARITY:e8bfe1eff2cc5839","status":"pending"}
14:24:15 [14:24:15.581]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11c7167910f0a029 epochNumber=1
{"provingJobId":"1:BASE_PARITY:11c7167910f0a029"}
14:24:15 [14:24:15.583]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testchA7IC/1 with map size 1073741824
14:24:15 [14:24:15.585]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.615]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.617]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45af4ae897271354 epochNumber=1
{"provingJobId":"1:BASE_PARITY:45af4ae897271354"}
14:24:15 [14:24:15.617]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:45af4ae897271354: Error: db error: Error: db error
14:24:15 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:45af4ae897271354"}
14:24:15 [14:24:15.621]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.621]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4dd79865be976da2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4dd79865be976da2"}
14:24:15 [14:24:15.623]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUCOekb/1 with map size 1073741824
14:24:15 [14:24:15.626]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.645]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4dd79865be976da2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4dd79865be976da2"}
14:24:15 [14:24:15.646]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4dd79865be976da2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4dd79865be976da2"}
14:24:15 [14:24:15.657]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.657]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff952bcf56432ad6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff952bcf56432ad6"}
14:24:15 [14:24:15.658]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1HaOro/1 with map size 1073741824
14:24:15 [14:24:15.660]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.680]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ff952bcf56432ad6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ff952bcf56432ad6"}
14:24:15 [14:24:15.680]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ff952bcf56432ad6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ff952bcf56432ad6"}
14:24:15 [14:24:15.681]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:ff952bcf56432ad6: Error: db error
14:24:15 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:ff952bcf56432ad6"}
14:24:15 [14:24:15.688]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.689]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5f11cc7c39ae5ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:c5f11cc7c39ae5ac"}
14:24:15 [14:24:15.691]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDMqp67/1 with map size 1073741824
14:24:15 [14:24:15.706]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.730]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c5f11cc7c39ae5ac type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c5f11cc7c39ae5ac"}
14:24:15 [14:24:15.731]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c5f11cc7c39ae5ac type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c5f11cc7c39ae5ac"}
14:24:15 [14:24:15.745]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.746]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1afc4d39d1f2d2c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1afc4d39d1f2d2c5"}
14:24:15 [14:24:15.748]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrIphsi/1 with map size 1073741824
14:24:15 [14:24:15.752]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.774]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1afc4d39d1f2d2c5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1afc4d39d1f2d2c5"}
14:24:15 [14:24:15.775]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1afc4d39d1f2d2c5 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1afc4d39d1f2d2c5"}
14:24:15 [14:24:15.776]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:1afc4d39d1f2d2c5 jobErr=test error: Error: db error
14:24:15 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:1afc4d39d1f2d2c5"}
14:24:15 [14:24:15.780]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.780]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1b6b2ca8c68d08dc not found
{"provingJobId":"1:BASE_PARITY:1b6b2ca8c68d08dc"}
14:24:15 [14:24:15.783]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.783]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6e8da028564ebebc err=test error
{"provingJoId":"1:BASE_PARITY:6e8da028564ebebc"}
14:24:15 [14:24:15.786]
INFO:
prover-client:proving-broker Proving Broker started
14:24:15 [14:24:15.800]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72cf3eadddceecba epochNumber=1
{"provingJobId":"1:BASE_PARITY:72cf3eadddceecba"}
14:24:15 [14:24:15.802]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf41JSk/1 with map size 1073741824
14:24:15 [14:24:15.804]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.824]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:72cf3eadddceecba type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:72cf3eadddceecba"}
14:24:15 [14:24:15.825]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:72cf3eadddceecba type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:72cf3eadddceecba"}
14:24:15 [14:24:15.845]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:4414950df0753f61 epochNumber=2
{"provingJobId":"2:BASE_PARITY:4414950df0753f61"}
14:24:15 [14:24:15.846]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testf41JSk/2 with map size 1073741824
14:24:15 [14:24:15.850]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.870]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:4414950df0753f61 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:4414950df0753f61"}
14:24:15 [14:24:15.871]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:4414950df0753f61 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:4414950df0753f61"}
14:24:15 [14:24:15.938]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:e3e322d58bec5ebd epochNumber=3
{"provingJobId":"3:BASE_PARITY:e3e322d58bec5ebd"}
14:24:15 [14:24:15.940]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testf41JSk/3 with map size 1073741824
14:24:15 [14:24:15.943]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:15 [14:24:15.964]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:15 [14:24:15.965]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:24:16 [14:24:16.025]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:f0427c2f3e6e6b04 epochNumber=4
{"provingJobId":"4:BASE_PARITY:f0427c2f3e6e6b04"}
14:24:16 [14:24:16.026]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testf41JSk/4 with map size 1073741824
14:24:16 [14:24:16.031]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:16 [14:24:16.071]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:16 [14:24:16.071]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
14:24:16 [14:24:16.116]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:61e6c45e1a5c7c02 epochNumber=5
{"provingJobId":"5:BASE_PARITY:61e6c45e1a5c7c02"}
14:24:16 [14:24:16.117]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testf41JSk/5 with map size 1073741824
14:24:16 [14:24:16.119]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:16 [14:24:16.151]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:16 [14:24:16.152]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
14:24:16
PASS src/proving_broker/proving_broker.test.ts (
13.585 s)
14:24:16 ProvingBroker
14:24:16 Producer API
14:24:16
✓ refuses stale jobs (52 ms)
14:24:16
✓ enqueues jobs (5 ms)
14:24:16
✓ ignores duplicate jobs (4 ms)
14:24:16
✓ reports correct status when enqueuing repeat jobs (5 ms)
14:24:16
✓ throws an error in case of duplicate job IDs (25 ms)
14:24:16
✓ returns not-found status for non-existing jobs (2 ms)
14:24:16
✓ cancels jobs in queue (3 ms)
14:24:16
✓ cancels jobs in-progress (2 ms)
14:24:16
✓ returns job result if successful (9 ms)
14:24:16
✓ returns job error if failed (3 ms)
14:24:16
✓ correctly returns job status for concurrent writes (3 ms)
14:24:16
✓ refuses stale jobs (33 ms)
14:24:16
✓ enqueues jobs (38 ms)
14:24:16
✓ ignores duplicate jobs (30 ms)
14:24:16
✓ reports correct status when enqueuing repeat jobs (31 ms)
14:24:16
✓ throws an error in case of duplicate job IDs (43 ms)
14:24:16
✓ returns not-found status for non-existing jobs (4 ms)
14:24:16
✓ cancels jobs in queue (30 ms)
14:24:16
✓ cancels jobs in-progress (32 ms)
14:24:16
✓ returns job result if successful (35 ms)
14:24:16
✓ returns job error if failed (33 ms)
14:24:16
✓ correctly returns job status for concurrent writes (41 ms)
14:24:16 Consumer API
14:24:16
✓ returns undefined if no jobs are available (1 ms)
14:24:16
✓ returns jobs in priority order (2 ms)
14:24:16
✓ returns undefined if no jobs are available for the given allowList (2 ms)
14:24:16
✓ returns a job if it is in the allowList (2 ms)
14:24:16
✓ returns the most important job if it is in the allowList (2 ms)
14:24:16
✓ returns any job if filter is empty (2 ms)
14:24:16
✓ returns a new job when reporting job success (3 ms)
14:24:16
✓ returns a new job when reporting permanent error (2 ms)
14:24:16
✓ returns a new job when reporting retry-able error (3 ms)
14:24:16
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
14:24:16
✓ returns a new job if job is already in progress elsewhere (1055 ms)
14:24:16
✓ avoids sending the same job to a new agent after a restart (1004 ms)
14:24:16
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
14:24:16
✓ tracks job result if in progress (3 ms)
14:24:16
✓ tracks job result even if job is in queue (2 ms)
14:24:16
✓ ignores reported job error if unknown job (1 ms)
14:24:16
✓ ignores job result if unknown job (1 ms)
14:24:16
✓ returns undefined if no jobs are available (3 ms)
14:24:16
✓ returns jobs in priority order (38 ms)
14:24:16
✓ returns undefined if no jobs are available for the given allowList (21 ms)
14:24:16
✓ returns a job if it is in the allowList (53 ms)
14:24:16
✓ returns the most important job if it is in the allowList (51 ms)
14:24:16
✓ returns any job if filter is empty (51 ms)
14:24:16
✓ returns a new job when reporting job success (37 ms)
14:24:16
✓ returns a new job when reporting permanent error (31 ms)
14:24:16
✓ returns a new job when reporting retry-able error (25 ms)
14:24:16
✓ returns a new job when reporting progress if current one is cancelled (32 ms)
14:24:16
✓ returns a new job if job is already in progress elsewhere (1108 ms)
14:24:16
✓ avoids sending the same job to a new agent after a restart (1062 ms)
14:24:16
✓ avoids sending a completed job to a new agent after a restart (1066 ms)
14:24:16
✓ tracks job result if in progress (58 ms)
14:24:16
✓ tracks job result even if job is in queue (64 ms)
14:24:16
✓ ignores reported job error if unknown job (5 ms)
14:24:16
✓ ignores job result if unknown job (5 ms)
14:24:16 Timeouts
14:24:16
✓ tracks in progress jobs (1 ms)
14:24:16
✓ re-enqueues jobs that time out (110 ms)
14:24:16
✓ cancel stale jobs that time out (237 ms)
14:24:16
✓ keeps the jobs in progress while it is alive (203 ms)
14:24:16
✓ tracks in progress jobs (29 ms)
14:24:16
✓ re-enqueues jobs that time out (136 ms)
14:24:16
✓ cancel stale jobs that time out (288 ms)
14:24:16
✓ keeps the jobs in progress while it is alive (247 ms)
14:24:16 Retries
14:24:16
✓ retries jobs (3 ms)
14:24:16
✓ retries up to a maximum number of times (2 ms)
14:24:16
✓ passing retry=false does not retry (1 ms)
14:24:16
✓ does not retry if job is stale (27 ms)
14:24:16
✓ retries jobs (23 ms)
14:24:16
✓ retries up to a maximum number of times (32 ms)
14:24:16
✓ passing retry=false does not retry (63 ms)
14:24:16
✓ does not retry if job is stale (97 ms)
14:24:16 Database management
14:24:16
✓ re-enqueues proof requests on start (3 ms)
14:24:16
✓ restores proof results on start (2 ms)
14:24:16
✓ only re-enqueues unfinished jobs (2 ms)
14:24:16
✓ saves job when enqueued (2 ms)
14:24:16
✓ does not retain job if database fails to save (2 ms)
14:24:16
✓ saves job result (2 ms)
14:24:16
✓ saves result even if database fails to save (3 ms)
14:24:16
✓ saves job error (2 ms)
14:24:16
✓ saves job error even if database fails to save (2 ms)
14:24:16
✓ does not save job result if job is unknown (2 ms)
14:24:16
✓ does not save job error if job is unknown (1 ms)
14:24:16
✓ cleans up old jobs periodically (262 ms)
14:24:16
✓ re-enqueues proof requests on start (75 ms)
14:24:16
✓ restores proof results on start (92 ms)
14:24:16
✓ only re-enqueues unfinished jobs (65 ms)
14:24:16
✓ saves job when enqueued (34 ms)
14:24:16
✓ does not retain job if database fails to save (4 ms)
14:24:16
✓ saves job result (35 ms)
14:24:16
✓ saves result even if database fails to save (30 ms)
14:24:16
✓ saves job error (56 ms)
14:24:16
✓ saves job error even if database fails to save (36 ms)
14:24:16
✓ does not save job result if job is unknown (4 ms)
14:24:16
✓ does not save job error if job is unknown (3 ms)
14:24:16
✓ cleans up old jobs periodically (405 ms)
14:24:16
14:24:16
Test Suites: 1 passed, 1 total
14:24:16
Tests: 96 passed, 96 total
14:24:16
Snapshots: 0 total
14:24:16
Time: 13.675 s
14:24:16
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
14:24:16
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?