Parent Log:
http://ci.aztec-labs.com/7d50fe3fd123434b
Command: 9e675ed54321cc5a yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/96a426245731b3f48fcb6dd5e2a683c858706602
Env: REF_NAME=gh-readonly-queue/next/pr-15083-5867520ed258adecfee692b3694d693b2c7964ca CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 16 21:53:45 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15083_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
21:53:50 [21:53:50.158]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.165]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08d4deed0befb919 epochNumber=42
{"provingJobId":"1:BASE_PARITY:08d4deed0befb919"}
21:53:50 [21:53:50.169]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:eb9f5e72d5f5b729 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb9f5e72d5f5b729"}
21:53:50 [21:53:50.196]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.197]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b833c3b7ab3aa07a epochNumber=1
{"provingJobId":"1:BASE_PARITY:b833c3b7ab3aa07a"}
21:53:50 [21:53:50.198]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e55e24c23ea7c32a epochNumber=1
{"provingJobId":"1:BASE_PARITY:e55e24c23ea7c32a"}
21:53:50 [21:53:50.200]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb5dc7644917de2a epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb5dc7644917de2a"}
21:53:50 [21:53:50.201]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:eb5dc7644917de2a epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:eb5dc7644917de2a"}
21:53:50 [21:53:50.204]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.204]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d056350e784f4e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d056350e784f4e6"}
21:53:50 [21:53:50.205]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0d056350e784f4e6 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0d056350e784f4e6"}
21:53:50 [21:53:50.206]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0d056350e784f4e6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0d056350e784f4e6"}
21:53:50 [21:53:50.206]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0d056350e784f4e6 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0d056350e784f4e6"}
21:53:50 [21:53:50.207]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a35d8f21cd61074 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a35d8f21cd61074"}
21:53:50 [21:53:50.232]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.234]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e1572d5cfcb7225 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e1572d5cfcb7225"}
21:53:50 [21:53:50.235]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7e1572d5cfcb7225
{"provingJobId":"1:BASE_PARITY:7e1572d5cfcb7225"}
21:53:50 [21:53:50.236]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7e1572d5cfcb7225 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7e1572d5cfcb7225"}
21:53:50 [21:53:50.236]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7e1572d5cfcb7225 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7e1572d5cfcb7225"}
21:53:50 [21:53:50.237]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:abfd746b26280433 epochNumber=1
{"provingJobId":"1:BASE_PARITY:abfd746b26280433"}
21:53:50 [21:53:50.239]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:abfd746b26280433
{"provingJobId":"1:BASE_PARITY:abfd746b26280433"}
21:53:50 [21:53:50.239]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:abfd746b26280433 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:abfd746b26280433"}
21:53:50 [21:53:50.240]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.241]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24e7449f6f87f3cf epochNumber=1
{"provingJobId":"1:BASE_PARITY:24e7449f6f87f3cf"}
21:53:50 [21:53:50.241]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:24e7449f6f87f3cf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:24e7449f6f87f3cf"}
21:53:50 [21:53:50.241]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:24e7449f6f87f3cf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:24e7449f6f87f3cf"}
21:53:50 [21:53:50.250]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.251]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d1c03a9a4e5e77f epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d1c03a9a4e5e77f"}
21:53:50 [21:53:50.251]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9d1c03a9a4e5e77f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9d1c03a9a4e5e77f"}
21:53:50 [21:53:50.252]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9d1c03a9a4e5e77f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9d1c03a9a4e5e77f"}
21:53:50 [21:53:50.253]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea72c1f56e00e6c3 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ea72c1f56e00e6c3"}
21:53:50 [21:53:50.254]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ea72c1f56e00e6c3 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ea72c1f56e00e6c3"}
21:53:50 [21:53:50.254]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:621a3ed02c4c07ea epochNumber=0
{"provingJobId":"1:BASE_PARITY:621a3ed02c4c07ea"}
21:53:50 [21:53:50.254]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ea72c1f56e00e6c3 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ea72c1f56e00e6c3"}
21:53:50 [21:53:50.254]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b2e8f4199e554e0 epochNumber=0
{"provingJobId":"1:BASE_PARITY:6b2e8f4199e554e0"}
21:53:50 [21:53:50.256]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.257]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.258]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b7320a50b9392ff epochNumber=2
{"provingJobId":"1:BASE_PARITY:1b7320a50b9392ff"}
21:53:50 [21:53:50.258]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2bb31d6fa0bc7f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2bb31d6fa0bc7f2"}
21:53:50 [21:53:50.260]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.260]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f9059a5658b0a6aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:f9059a5658b0a6aa"}
21:53:50 [21:53:50.261]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.262]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fea16e7fb093ed23 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fea16e7fb093ed23"}
21:53:50 [21:53:50.262]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:461eba0be34c200e epochNumber=1
{"provingJobId":"1:BASE_PARITY:461eba0be34c200e"}
21:53:50 [21:53:50.263]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:313cfa6f6a5b3458 epochNumber=2
{"provingJobId":"1:BASE_PARITY:313cfa6f6a5b3458"}
21:53:50 [21:53:50.263]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86ee4dad1fb071ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:86ee4dad1fb071ea"}
21:53:50 [21:53:50.264]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:716cf2f1b7180df3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:716cf2f1b7180df3"}
21:53:50 [21:53:50.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f85fb6d732b67d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f85fb6d732b67d3"}
21:53:50 [21:53:50.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42dede3cc3b9ff7f epochNumber=2
{"provingJobId":"1:BASE_PARITY:42dede3cc3b9ff7f"}
21:53:50 [21:53:50.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f5f7829ca291a96 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f5f7829ca291a96"}
21:53:50 [21:53:50.268]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4cb1fd2d47185163 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4cb1fd2d47185163"}
21:53:50 [21:53:50.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed64bef958bdff1d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed64bef958bdff1d"}
21:53:50 [21:53:50.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7c863b9c7be4df2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e7c863b9c7be4df2"}
21:53:50 [21:53:50.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a05b9e4a923fc744 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a05b9e4a923fc744"}
21:53:50 [21:53:50.271]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.271]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff64a00439be69c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff64a00439be69c2"}
21:53:50 [21:53:50.272]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e75a577d70d8dc0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:e75a577d70d8dc0e"}
21:53:50 [21:53:50.272]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ff64a00439be69c2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ff64a00439be69c2"}
21:53:50 [21:53:50.273]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.274]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:275a8c14a4022b99 epochNumber=1
{"provingJobId":"1:BASE_PARITY:275a8c14a4022b99"}
21:53:50 [21:53:50.274]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1201156c4bf15802 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1201156c4bf15802"}
21:53:50 [21:53:50.274]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:275a8c14a4022b99 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:275a8c14a4022b99"}
21:53:50 [21:53:50.275]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.276]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d31e54062f64d597 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d31e54062f64d597"}
21:53:50 [21:53:50.276]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a91f56efe980492d epochNumber=1
{"provingJobId":"1:BASE_PARITY:a91f56efe980492d"}
21:53:50 [21:53:50.277]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d31e54062f64d597 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:d31e54062f64d597"}
21:53:50 [21:53:50.277]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:733b3489603a5ad2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:733b3489603a5ad2"}
21:53:50 [21:53:50.280]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:733b3489603a5ad2
{"provingJobId":"1:BASE_PARITY:733b3489603a5ad2"}
21:53:50 [21:53:50.280]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:733b3489603a5ad2 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:733b3489603a5ad2"}
21:53:50 [21:53:50.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6746bb8161aca0f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6746bb8161aca0f9"}
21:53:50 [21:53:50.282]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:733b3489603a5ad2 has already been completed
{"provingJobId":"1:BASE_PARITY:733b3489603a5ad2"}
21:53:50 [21:53:50.283]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf8e56910c507eeb epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf8e56910c507eeb"}
21:53:50 [21:53:50.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76415555dff93c77 epochNumber=2
{"provingJobId":"1:BASE_PARITY:76415555dff93c77"}
21:53:51 [21:53:51.336]
INFO:
prover-client:proving-broker Proving Broker started
21:53:51 [21:53:51.337]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bf8e56910c507eeb settled=false
{"provingJobId":"1:BASE_PARITY:bf8e56910c507eeb","status":"pending"}
21:53:51 [21:53:51.338]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:76415555dff93c77 settled=false
{"provingJobId":"1:BASE_PARITY:76415555dff93c77","status":"pending"}
21:53:51 [21:53:51.339]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bf8e56910c507eeb type=BASE_PARITY startedAt=1750110830284 older agent has taken job
{"provingJobId":"1:BASE_PARITY:bf8e56910c507eeb"}
21:53:51 [21:53:51.339]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bf8e56910c507eeb type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:bf8e56910c507eeb"}
21:53:51 [21:53:51.341]
INFO:
prover-client:proving-broker Proving Broker started
21:53:51 [21:53:51.341]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71c36b35cf73d13a epochNumber=1
{"provingJobId":"1:BASE_PARITY:71c36b35cf73d13a"}
21:53:51 [21:53:51.342]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9274a47e989d0a09 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9274a47e989d0a09"}
21:53:52 [21:53:52.343]
INFO:
prover-client:proving-broker Proving Broker started
21:53:52 [21:53:52.344]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:71c36b35cf73d13a settled=false
{"provingJobId":"1:BASE_PARITY:71c36b35cf73d13a","status":"pending"}
21:53:52 [21:53:52.344]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9274a47e989d0a09 settled=false
{"provingJobId":"1:BASE_PARITY:9274a47e989d0a09","status":"pending"}
21:53:52 [21:53:52.345]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:71c36b35cf73d13a type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:71c36b35cf73d13a"}
21:53:52 [21:53:52.347]
INFO:
prover-client:proving-broker Proving Broker started
21:53:52 [21:53:52.348]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be8bb4bc96ce03c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:be8bb4bc96ce03c4"}
21:53:52 [21:53:52.348]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a365109a11190c14 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a365109a11190c14"}
21:53:53 [21:53:53.350]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.350]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:be8bb4bc96ce03c4 settled=false
{"provingJobId":"1:BASE_PARITY:be8bb4bc96ce03c4","status":"pending"}
21:53:53 [21:53:53.350]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a365109a11190c14 settled=false
{"provingJobId":"1:BASE_PARITY:a365109a11190c14","status":"pending"}
21:53:53 [21:53:53.351]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:be8bb4bc96ce03c4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:be8bb4bc96ce03c4"}
21:53:53 [21:53:53.351]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:be8bb4bc96ce03c4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:be8bb4bc96ce03c4"}
21:53:53 [21:53:53.356]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4774aca6282cfc94 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4774aca6282cfc94"}
21:53:53 [21:53:53.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:39bd98a67314bbe7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:39bd98a67314bbe7"}
21:53:53 [21:53:53.358]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4774aca6282cfc94 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4774aca6282cfc94"}
21:53:53 [21:53:53.359]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:39bd98a67314bbe7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:39bd98a67314bbe7"}
21:53:53 [21:53:53.360]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.363]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17e6b1ec04a086e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17e6b1ec04a086e8"}
21:53:53 [21:53:53.364]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c695785e8ca68b9e epochNumber=2
{"provingJobId":"1:BASE_PARITY:c695785e8ca68b9e"}
21:53:53 [21:53:53.364]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:17e6b1ec04a086e8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:17e6b1ec04a086e8"}
21:53:53 [21:53:53.365]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:17e6b1ec04a086e8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:17e6b1ec04a086e8"}
21:53:53 [21:53:53.365]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c695785e8ca68b9e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c695785e8ca68b9e"}
21:53:53 [21:53:53.366]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.367]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ca331140794aa056 err=test error
{"provingJoId":"1:BASE_PARITY:ca331140794aa056"}
21:53:53 [21:53:53.368]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.368]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d2b29ef4159a1c7e not found
{"provingJobId":"1:BASE_PARITY:d2b29ef4159a1c7e"}
21:53:53 [21:53:53.370]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.370]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5c8f6451f4209ef epochNumber=1
{"provingJobId":"1:BASE_PARITY:c5c8f6451f4209ef"}
21:53:53 [21:53:53.372]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.372]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c8dae5a44dbccc4d epochNumber=1
{"provingJobId":"1:BASE_PARITY:c8dae5a44dbccc4d"}
21:53:53 [21:53:53.474]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c8dae5a44dbccc4d timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c8dae5a44dbccc4d"}
21:53:53 [21:53:53.481]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3be6bcb5830d711 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e3be6bcb5830d711"}
21:53:53 [21:53:53.585]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e3be6bcb5830d711 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e3be6bcb5830d711"}
21:53:53 [21:53:53.590]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2514076bee7e0fd epochNumber=10
{"provingJobId":"1:BASE_PARITY:b2514076bee7e0fd"}
21:53:53 [21:53:53.611]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:53 [21:53:53.718]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.719]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8afee22173bb31b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e8afee22173bb31b"}
21:53:53 [21:53:53.870]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e8afee22173bb31b timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e8afee22173bb31b"}
21:53:53 [21:53:53.923]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.923]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f60f337f0fd1c67c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f60f337f0fd1c67c"}
21:53:53 [21:53:53.924]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f60f337f0fd1c67c type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f60f337f0fd1c67c"}
21:53:53 [21:53:53.925]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.926]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:717cc86ac38cd7d8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:717cc86ac38cd7d8"}
21:53:53 [21:53:53.926]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:717cc86ac38cd7d8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:717cc86ac38cd7d8"}
21:53:53 [21:53:53.927]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:717cc86ac38cd7d8 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:717cc86ac38cd7d8"}
21:53:53 [21:53:53.928]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.928]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16ae11e41c355265 epochNumber=1
{"provingJobId":"1:BASE_PARITY:16ae11e41c355265"}
21:53:53 [21:53:53.929]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:16ae11e41c355265 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:16ae11e41c355265"}
21:53:53 [21:53:53.929]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.930]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab0317b46db64ab6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab0317b46db64ab6"}
21:53:53 [21:53:53.930]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ab0317b46db64ab6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:ab0317b46db64ab6"}
21:53:53 [21:53:53.931]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36c9e2a6fadc2d99 epochNumber=3
{"provingJobId":"1:BASE_PARITY:36c9e2a6fadc2d99"}
21:53:53 [21:53:53.955]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:53 [21:53:53.957]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ab0317b46db64ab6 err=test error
{"provingJoId":"1:BASE_PARITY:ab0317b46db64ab6"}
21:53:53 [21:53:53.959]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.960]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d0166cdd99d1ccbd settled=false
{"provingJobId":"1:BASE_PARITY:d0166cdd99d1ccbd","status":"pending"}
21:53:53 [21:53:53.960]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a6f2815cb4b70dac settled=false
{"provingJobId":"1:BASE_PARITY:a6f2815cb4b70dac","status":"pending"}
21:53:53 [21:53:53.963]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.963]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:310e7496ae16d2c6 settled=true
{"provingJobId":"1:BASE_PARITY:310e7496ae16d2c6","status":"fulfilled"}
21:53:53 [21:53:53.963]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:2806820cfa5657fa settled=true
{"provingJobId":"2:BASE_PARITY:2806820cfa5657fa","status":"fulfilled"}
21:53:53 [21:53:53.965]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.965]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6e1fe8778c31beaf settled=true
{"provingJobId":"1:BASE_PARITY:6e1fe8778c31beaf","status":"fulfilled"}
21:53:53 [21:53:53.965]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:337e6598ee5c5e04 settled=false
{"provingJobId":"1:BASE_PARITY:337e6598ee5c5e04","status":"pending"}
21:53:53 [21:53:53.966]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.968]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d359167cdbbc8a9d epochNumber=1
{"provingJobId":"1:BASE_PARITY:d359167cdbbc8a9d"}
21:53:53 [21:53:53.970]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.970]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e86d71e3e8a632b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e86d71e3e8a632b9"}
21:53:53 [21:53:53.971]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:e86d71e3e8a632b9: Error: db error: Error: db error
21:53:53 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:e86d71e3e8a632b9"}
21:53:53 [21:53:53.972]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.973]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68cab930d14225c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:68cab930d14225c6"}
21:53:53 [21:53:53.973]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:68cab930d14225c6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:68cab930d14225c6"}
21:53:53 [21:53:53.973]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:68cab930d14225c6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:68cab930d14225c6"}
21:53:53 [21:53:53.974]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.975]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:109944b5357c6737 epochNumber=1
{"provingJobId":"1:BASE_PARITY:109944b5357c6737"}
21:53:53 [21:53:53.975]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:109944b5357c6737 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:109944b5357c6737"}
21:53:53 [21:53:53.975]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:109944b5357c6737 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:109944b5357c6737"}
21:53:53 [21:53:53.976]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:109944b5357c6737: Error: db error
21:53:53 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:109944b5357c6737"}
21:53:53 [21:53:53.977]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.978]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:462ac91a3abff7ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:462ac91a3abff7ca"}
21:53:53 [21:53:53.978]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:462ac91a3abff7ca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:462ac91a3abff7ca"}
21:53:53 [21:53:53.978]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:462ac91a3abff7ca type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:462ac91a3abff7ca"}
21:53:53 [21:53:53.980]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.980]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a7943910900c80de epochNumber=1
{"provingJobId":"1:BASE_PARITY:a7943910900c80de"}
21:53:53 [21:53:53.980]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a7943910900c80de type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a7943910900c80de"}
21:53:53 [21:53:53.980]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a7943910900c80de type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a7943910900c80de"}
21:53:53 [21:53:53.981]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a7943910900c80de jobErr=test error: Error: db error
21:53:53 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:a7943910900c80de"}
21:53:53 [21:53:53.982]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.983]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fca85561848e4a02 not found
{"provingJobId":"1:BASE_PARITY:fca85561848e4a02"}
21:53:53 [21:53:53.985]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.985]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:325b57ccc4ff638c err=test error
{"provingJoId":"1:BASE_PARITY:325b57ccc4ff638c"}
21:53:53 [21:53:53.987]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.998]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f98001280e9220d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f98001280e9220d5"}
21:53:53 [21:53:53.998]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f98001280e9220d5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f98001280e9220d5"}
21:53:53 [21:53:53.998]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f98001280e9220d5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f98001280e9220d5"}
21:53:54 [21:53:54.010]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:f6aced05741260cb epochNumber=2
{"provingJobId":"2:BASE_PARITY:f6aced05741260cb"}
21:53:54 [21:53:54.011]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:f6aced05741260cb type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:f6aced05741260cb"}
21:53:54 [21:53:54.012]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:f6aced05741260cb type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:f6aced05741260cb"}
21:53:54 [21:53:54.074]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:d4feccd248da157c epochNumber=3
{"provingJobId":"3:BASE_PARITY:d4feccd248da157c"}
21:53:54 [21:53:54.091]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:54 [21:53:54.136]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:12b75e2dc8c7c624 epochNumber=4
{"provingJobId":"4:BASE_PARITY:12b75e2dc8c7c624"}
21:53:54 [21:53:54.144]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:54 [21:53:54.200]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:409547924d3a519c epochNumber=5
{"provingJobId":"5:BASE_PARITY:409547924d3a519c"}
21:53:54 [21:53:54.221]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:54 [21:53:54.256]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.257]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7cdbc3abaa02eccd epochNumber=42
{"provingJobId":"1:BASE_PARITY:7cdbc3abaa02eccd"}
21:53:54 [21:53:54.259]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testM5QCzN/42 with map size 1073741824
21:53:54 [21:53:54.269]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.295]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:0ec445dc8f966956 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0ec445dc8f966956"}
21:53:54 [21:53:54.300]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.301]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e251d6a74c101bee epochNumber=1
{"provingJobId":"1:BASE_PARITY:e251d6a74c101bee"}
21:53:54 [21:53:54.302]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDGrfg4/1 with map size 1073741824
21:53:54 [21:53:54.305]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.323]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:38dece315da4e1ef epochNumber=1
{"provingJobId":"1:BASE_PARITY:38dece315da4e1ef"}
21:53:54 [21:53:54.332]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.332]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:803f4bf95ff0ced7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:803f4bf95ff0ced7"}
21:53:54 [21:53:54.333]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testq2bGnC/1 with map size 1073741824
21:53:54 [21:53:54.336]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.359]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:803f4bf95ff0ced7 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:803f4bf95ff0ced7"}
21:53:54 [21:53:54.363]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.364]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:62a7d7ae118b6ace epochNumber=1
{"provingJobId":"1:BASE_PARITY:62a7d7ae118b6ace"}
21:53:54 [21:53:54.365]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-teststcsOe/1 with map size 1073741824
21:53:54 [21:53:54.367]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.384]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:62a7d7ae118b6ace epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:62a7d7ae118b6ace"}
21:53:54 [21:53:54.385]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:62a7d7ae118b6ace type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:62a7d7ae118b6ace"}
21:53:54 [21:53:54.394]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:62a7d7ae118b6ace epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:62a7d7ae118b6ace"}
21:53:54 [21:53:54.398]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.399]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43534bc68529b8c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43534bc68529b8c7"}
21:53:54 [21:53:54.400]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOrNtOc/1 with map size 1073741824
21:53:54 [21:53:54.402]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.427]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.432]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.433]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a4e5b2bf2ef886c epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a4e5b2bf2ef886c"}
21:53:54 [21:53:54.433]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBFGryW/1 with map size 1073741824
21:53:54 [21:53:54.435]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.454]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7a4e5b2bf2ef886c
{"provingJobId":"1:BASE_PARITY:7a4e5b2bf2ef886c"}
21:53:54 [21:53:54.455]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7a4e5b2bf2ef886c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7a4e5b2bf2ef886c"}
21:53:54 [21:53:54.455]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7a4e5b2bf2ef886c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7a4e5b2bf2ef886c"}
21:53:54 [21:53:54.467]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.468]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52de670353406934 epochNumber=1
{"provingJobId":"1:BASE_PARITY:52de670353406934"}
21:53:54 [21:53:54.471]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testl8s4RA/1 with map size 1073741824
21:53:54 [21:53:54.474]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.497]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:52de670353406934
{"provingJobId":"1:BASE_PARITY:52de670353406934"}
21:53:54 [21:53:54.497]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:52de670353406934 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:52de670353406934"}
21:53:54 [21:53:54.508]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.509]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84cf6ce15f33b7b7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:84cf6ce15f33b7b7"}
21:53:54 [21:53:54.510]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeXTllV/1 with map size 1073741824
21:53:54 [21:53:54.512]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.531]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:84cf6ce15f33b7b7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:84cf6ce15f33b7b7"}
21:53:54 [21:53:54.532]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:84cf6ce15f33b7b7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:84cf6ce15f33b7b7"}
21:53:54 [21:53:54.541]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.542]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7da383a351e02647 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7da383a351e02647"}
21:53:54 [21:53:54.543]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjmPLuf/1 with map size 1073741824
21:53:54 [21:53:54.546]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.566]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7da383a351e02647 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7da383a351e02647"}
21:53:54 [21:53:54.567]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7da383a351e02647 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7da383a351e02647"}
21:53:54 [21:53:54.581]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d701de073d2d49a3 epochNumber=0
{"provingJobId":"1:BASE_PARITY:d701de073d2d49a3"}
21:53:54 [21:53:54.583]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test5s5he8/0 with map size 1073741824
21:53:54 [21:53:54.586]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.609]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d701de073d2d49a3 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d701de073d2d49a3"}
21:53:54 [21:53:54.610]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f23296c47ea02a9 epochNumber=0
{"provingJobId":"1:BASE_PARITY:8f23296c47ea02a9"}
21:53:54 [21:53:54.610]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d701de073d2d49a3 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d701de073d2d49a3"}
21:53:54 [21:53:54.610]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cfde3c98d782c6ff epochNumber=0
{"provingJobId":"1:BASE_PARITY:cfde3c98d782c6ff"}
21:53:54 [21:53:54.626]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.630]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.631]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:886d40dee84c7f1a epochNumber=2
{"provingJobId":"1:BASE_PARITY:886d40dee84c7f1a"}
21:53:54 [21:53:54.632]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testh8cU4E/2 with map size 1073741824
21:53:54 [21:53:54.634]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.651]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:39d507c3f4aa6db8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:39d507c3f4aa6db8"}
21:53:54 [21:53:54.654]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testh8cU4E/1 with map size 1073741824
21:53:54 [21:53:54.658]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.682]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a4235184f995ae4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9a4235184f995ae4"}
21:53:54 [21:53:54.684]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfr1Nir/1 with map size 1073741824
21:53:54 [21:53:54.686]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.709]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.710]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e43f0af415849ee5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e43f0af415849ee5"}
21:53:54 [21:53:54.711]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2IFTvu/1 with map size 1073741824
21:53:54 [21:53:54.715]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.733]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc624d2f6793cee4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc624d2f6793cee4"}
21:53:54 [21:53:54.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7bb8efd35c4d92ab epochNumber=2
{"provingJobId":"1:BASE_PARITY:7bb8efd35c4d92ab"}
21:53:54 [21:53:54.741]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2IFTvu/2 with map size 1073741824
21:53:54 [21:53:54.743]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.766]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:889f480912f6d6d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:889f480912f6d6d9"}
21:53:54 [21:53:54.777]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ec6d82ebc60fbb4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ec6d82ebc60fbb4"}
21:53:54 [21:53:54.778]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtnQi03/1 with map size 1073741824
21:53:54 [21:53:54.781]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1136270e3d16c994 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1136270e3d16c994"}
21:53:54 [21:53:54.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8f6f8298b3811cb epochNumber=2
{"provingJobId":"1:BASE_PARITY:f8f6f8298b3811cb"}
21:53:54 [21:53:54.814]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtnQi03/2 with map size 1073741824
21:53:54 [21:53:54.818]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.835]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:321bc036f01e95c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:321bc036f01e95c3"}
21:53:54 [21:53:54.845]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1f7b7b7570a0808 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1f7b7b7570a0808"}
21:53:54 [21:53:54.847]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testd6uqw8/1 with map size 1073741824
21:53:54 [21:53:54.851]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.882]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cedc39155f3268d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cedc39155f3268d6"}
21:53:54 [21:53:54.888]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78e66927c0bb7f9a epochNumber=2
{"provingJobId":"1:BASE_PARITY:78e66927c0bb7f9a"}
21:53:54 [21:53:54.889]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testd6uqw8/2 with map size 1073741824
21:53:54 [21:53:54.891]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.908]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7664343950aa52d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7664343950aa52d"}
21:53:54 [21:53:54.922]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.923]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2fa837126974208 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d2fa837126974208"}
21:53:54 [21:53:54.924]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testn5JBQl/1 with map size 1073741824
21:53:54 [21:53:54.925]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc0f6d6d0cf86650 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bc0f6d6d0cf86650"}
21:53:54 [21:53:54.950]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d2fa837126974208 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d2fa837126974208"}
21:53:54 [21:53:54.969]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.970]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f62c9ef11e7ad0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f62c9ef11e7ad0e"}
21:53:54 [21:53:54.971]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiDpjJP/1 with map size 1073741824
21:53:54 [21:53:54.974]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.991]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1bfa05de34ff71f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1bfa05de34ff71f"}
21:53:54 [21:53:54.998]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4f62c9ef11e7ad0e type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:4f62c9ef11e7ad0e"}
21:53:55 [21:53:55.007]
INFO:
prover-client:proving-broker Proving Broker started
21:53:55 [21:53:55.008]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6dc7bc519eea024b epochNumber=1
{"provingJobId":"1:BASE_PARITY:6dc7bc519eea024b"}
21:53:55 [21:53:55.008]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testccyjHX/1 with map size 1073741824
21:53:55 [21:53:55.010]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:55 [21:53:55.027]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b92714e17efbdc3a epochNumber=1
{"provingJobId":"1:BASE_PARITY:b92714e17efbdc3a"}
21:53:55 [21:53:55.033]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6dc7bc519eea024b type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:6dc7bc519eea024b"}
21:53:55 [21:53:55.037]
INFO:
prover-client:proving-broker Proving Broker started
21:53:55 [21:53:55.037]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43493af05ca81da5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43493af05ca81da5"}
21:53:55 [21:53:55.038]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1b8hBK/1 with map size 1073741824
21:53:55 [21:53:55.040]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:55 [21:53:55.057]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:43493af05ca81da5
{"provingJobId":"1:BASE_PARITY:43493af05ca81da5"}
21:53:55 [21:53:55.057]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:43493af05ca81da5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:43493af05ca81da5"}
21:53:55 [21:53:55.065]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5926edb9fb67235d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5926edb9fb67235d"}
21:53:55 [21:53:55.071]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:43493af05ca81da5 has already been completed
{"provingJobId":"1:BASE_PARITY:43493af05ca81da5"}
21:53:55 [21:53:55.076]
INFO:
prover-client:proving-broker Proving Broker started
21:53:55 [21:53:55.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0de7e060f640843f epochNumber=1
{"provingJobId":"1:BASE_PARITY:0de7e060f640843f"}
21:53:55 [21:53:55.078]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1JFpJG/1 with map size 1073741824
21:53:55 [21:53:55.081]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:55 [21:53:55.096]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:efaf17e157431743 epochNumber=2
{"provingJobId":"1:BASE_PARITY:efaf17e157431743"}
21:53:55 [21:53:55.097]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test1JFpJG/2 with map size 1073741824
21:53:55 [21:53:55.102]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:56 [21:53:56.172]
INFO:
prover-client:proving-broker Proving Broker started
21:53:56 [21:53:56.178]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0de7e060f640843f settled=false
{"provingJobId":"1:BASE_PARITY:0de7e060f640843f","status":"pending"}
21:53:56 [21:53:56.180]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:efaf17e157431743 settled=false
{"provingJobId":"1:BASE_PARITY:efaf17e157431743","status":"pending"}
21:53:56 [21:53:56.181]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0de7e060f640843f type=BASE_PARITY startedAt=1750110835119 older agent has taken job
{"provingJobId":"1:BASE_PARITY:0de7e060f640843f"}
21:53:56 [21:53:56.182]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0de7e060f640843f type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:0de7e060f640843f"}
21:53:56 [21:53:56.185]
INFO:
prover-client:proving-broker Proving Broker started
21:53:56 [21:53:56.186]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7a07970b771d398 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7a07970b771d398"}
21:53:56 [21:53:56.187]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7BONez/1 with map size 1073741824
21:53:56 [21:53:56.189]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:56 [21:53:56.207]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c84a61b338019ff0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c84a61b338019ff0"}
21:53:56 [21:53:56.210]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7BONez/2 with map size 1073741824
21:53:56 [21:53:56.212]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:57 [21:53:57.231]
INFO:
prover-client:proving-broker Proving Broker started
21:53:57 [21:53:57.233]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b7a07970b771d398 settled=false
{"provingJobId":"1:BASE_PARITY:b7a07970b771d398","status":"pending"}
21:53:57 [21:53:57.235]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c84a61b338019ff0 settled=false
{"provingJobId":"1:BASE_PARITY:c84a61b338019ff0","status":"pending"}
21:53:57 [21:53:57.236]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b7a07970b771d398 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:b7a07970b771d398"}
21:53:57 [21:53:57.241]
INFO:
prover-client:proving-broker Proving Broker started
21:53:57 [21:53:57.242]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f737a3c85d832f59 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f737a3c85d832f59"}
21:53:57 [21:53:57.243]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUg9tfO/1 with map size 1073741824
21:53:57 [21:53:57.245]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:57 [21:53:57.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5332ac8355c2cb1d epochNumber=2
{"provingJobId":"1:BASE_PARITY:5332ac8355c2cb1d"}
21:53:57 [21:53:57.273]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUg9tfO/2 with map size 1073741824
21:53:57 [21:53:57.279]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:58 [21:53:58.297]
INFO:
prover-client:proving-broker Proving Broker started
21:53:58 [21:53:58.300]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f737a3c85d832f59 settled=false
{"provingJobId":"1:BASE_PARITY:f737a3c85d832f59","status":"pending"}
21:53:58 [21:53:58.302]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5332ac8355c2cb1d settled=false
{"provingJobId":"1:BASE_PARITY:5332ac8355c2cb1d","status":"pending"}
21:53:58 [21:53:58.303]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f737a3c85d832f59 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f737a3c85d832f59"}
21:53:58 [21:53:58.303]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f737a3c85d832f59 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f737a3c85d832f59"}
21:53:58 [21:53:58.314]
INFO:
prover-client:proving-broker Proving Broker started
21:53:58 [21:53:58.315]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a325783089042476 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a325783089042476"}
21:53:58 [21:53:58.318]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testA3eIQE/1 with map size 1073741824
21:53:58 [21:53:58.319]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:58 [21:53:58.337]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34aeb5da4302bc79 epochNumber=2
{"provingJobId":"1:BASE_PARITY:34aeb5da4302bc79"}
21:53:58 [21:53:58.339]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testA3eIQE/2 with map size 1073741824
21:53:58 [21:53:58.341]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:58 [21:53:58.361]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a325783089042476 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a325783089042476"}
21:53:58 [21:53:58.370]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:34aeb5da4302bc79 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:34aeb5da4302bc79"}
21:53:58 [21:53:58.380]
INFO:
prover-client:proving-broker Proving Broker started
21:53:58 [21:53:58.381]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c20adea9538dc3d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c20adea9538dc3d0"}
21:53:58 [21:53:58.382]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOswtzd/1 with map size 1073741824
21:53:58 [21:53:58.386]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:58 [21:53:58.404]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbd5231553bc22e1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dbd5231553bc22e1"}
21:53:58 [21:53:58.406]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOswtzd/2 with map size 1073741824
21:53:58 [21:53:58.408]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:58 [21:53:58.424]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c20adea9538dc3d0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c20adea9538dc3d0"}
21:53:58 [21:53:58.424]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c20adea9538dc3d0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c20adea9538dc3d0"}
21:53:58 [21:53:58.430]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dbd5231553bc22e1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:dbd5231553bc22e1"}
21:53:58 [21:53:58.440]
INFO:
prover-client:proving-broker Proving Broker started
21:53:58 [21:53:58.441]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3734e6129fda58b0 err=test error
{"provingJoId":"1:BASE_PARITY:3734e6129fda58b0"}
21:53:58 [21:53:58.445]
INFO:
prover-client:proving-broker Proving Broker started
21:53:58 [21:53:58.446]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:591a67460d054366 not found
{"provingJobId":"1:BASE_PARITY:591a67460d054366"}
21:53:58 [21:53:58.450]
INFO:
prover-client:proving-broker Proving Broker started
21:53:58 [21:53:58.451]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e2548d8457cf3d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e2548d8457cf3d0"}
21:53:58 [21:53:58.451]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testM57O0X/1 with map size 1073741824
21:53:58 [21:53:58.454]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:58 [21:53:58.478]
INFO:
prover-client:proving-broker Proving Broker started
21:53:58 [21:53:58.479]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a289073a96f39b20 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a289073a96f39b20"}
21:53:58 [21:53:58.479]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testatqA2t/1 with map size 1073741824
21:53:58 [21:53:58.483]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:58 [21:53:58.609]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a289073a96f39b20 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a289073a96f39b20"}
21:53:58 [21:53:58.615]
INFO:
prover-client:proving-broker Proving Broker started
21:53:58 [21:53:58.616]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e367bc6e349ba56 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e367bc6e349ba56"}
21:53:58 [21:53:58.618]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTpiM1f/1 with map size 1073741824
21:53:58 [21:53:58.621]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:58 [21:53:58.768]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4e367bc6e349ba56 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4e367bc6e349ba56"}
21:53:58 [21:53:58.771]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ac20d671987288b8 epochNumber=10
{"provingJobId":"1:BASE_PARITY:ac20d671987288b8"}
21:53:58 [21:53:58.772]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testTpiM1f/10 with map size 1073741824
21:53:58 [21:53:58.776]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:58 [21:53:58.819]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:58 [21:53:58.819]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:53:58 [21:53:58.925]
INFO:
prover-client:proving-broker Proving Broker started
21:53:58 [21:53:58.926]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4433ce11f8e4810 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e4433ce11f8e4810"}
21:53:58 [21:53:58.927]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtFdSOL/1 with map size 1073741824
21:53:58 [21:53:58.931]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.132]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e4433ce11f8e4810 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e4433ce11f8e4810"}
21:53:59 [21:53:59.179]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.179]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53d14c1dec86d739 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53d14c1dec86d739"}
21:53:59 [21:53:59.180]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgv4KKz/1 with map size 1073741824
21:53:59 [21:53:59.182]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.202]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:53d14c1dec86d739 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:53d14c1dec86d739"}
21:53:59 [21:53:59.208]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.210]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99c2b08c6f39e340 epochNumber=1
{"provingJobId":"1:BASE_PARITY:99c2b08c6f39e340"}
21:53:59 [21:53:59.211]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlk7P1v/1 with map size 1073741824
21:53:59 [21:53:59.213]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.232]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:99c2b08c6f39e340 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:99c2b08c6f39e340"}
21:53:59 [21:53:59.234]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:99c2b08c6f39e340 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:99c2b08c6f39e340"}
21:53:59 [21:53:59.247]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.248]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc8a9afd178a7c1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc8a9afd178a7c1f"}
21:53:59 [21:53:59.250]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0eHnzI/1 with map size 1073741824
21:53:59 [21:53:59.255]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.274]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cc8a9afd178a7c1f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:cc8a9afd178a7c1f"}
21:53:59 [21:53:59.287]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.288]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5918a33a57e59fa8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5918a33a57e59fa8"}
21:53:59 [21:53:59.290]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzSL0X7/1 with map size 1073741824
21:53:59 [21:53:59.292]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.312]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5918a33a57e59fa8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:5918a33a57e59fa8"}
21:53:59 [21:53:59.314]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:feee5081cbed062e epochNumber=3
{"provingJobId":"1:BASE_PARITY:feee5081cbed062e"}
21:53:59 [21:53:59.315]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testzSL0X7/3 with map size 1073741824
21:53:59 [21:53:59.320]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.364]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:59 [21:53:59.365]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:53:59 [21:53:59.366]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:5918a33a57e59fa8 err=test error
{"provingJoId":"1:BASE_PARITY:5918a33a57e59fa8"}
21:53:59 [21:53:59.370]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testphuiTR/1 with map size 1073741824
21:53:59 [21:53:59.372]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.390]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testphuiTR/2 with map size 1073741824
21:53:59 [21:53:59.394]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.417]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.420]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:adad6b91d34debc7 settled=false
{"provingJobId":"1:BASE_PARITY:adad6b91d34debc7","status":"pending"}
21:53:59 [21:53:59.421]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:564408c9a00c91e8 settled=false
{"provingJobId":"1:BASE_PARITY:564408c9a00c91e8","status":"pending"}
21:53:59 [21:53:59.432]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testekDqts/1 with map size 1073741824
21:53:59 [21:53:59.436]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.459]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testekDqts/2 with map size 1073741824
21:53:59 [21:53:59.461]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.493]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.496]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:392bd2008923366f settled=true
{"provingJobId":"1:BASE_PARITY:392bd2008923366f","status":"fulfilled"}
21:53:59 [21:53:59.498]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:9627421e915e06e2 settled=true
{"provingJobId":"2:BASE_PARITY:9627421e915e06e2","status":"fulfilled"}
21:53:59 [21:53:59.506]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtNhLjN/1 with map size 1073741824
21:53:59 [21:53:59.510]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.541]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtNhLjN/2 with map size 1073741824
21:53:59 [21:53:59.545]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.565]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.567]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a64f9b8630611f7b settled=true
{"provingJobId":"1:BASE_PARITY:a64f9b8630611f7b","status":"fulfilled"}
21:53:59 [21:53:59.569]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:72e98249a129e2d2 settled=false
{"provingJobId":"1:BASE_PARITY:72e98249a129e2d2","status":"pending"}
21:53:59 [21:53:59.576]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.577]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:203e4a0a74cc6a2c epochNumber=1
{"provingJobId":"1:BASE_PARITY:203e4a0a74cc6a2c"}
21:53:59 [21:53:59.578]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0YiddQ/1 with map size 1073741824
21:53:59 [21:53:59.582]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.607]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40d7995935b8455a epochNumber=1
{"provingJobId":"1:BASE_PARITY:40d7995935b8455a"}
21:53:59 [21:53:59.609]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:40d7995935b8455a: Error: db error: Error: db error
21:53:59 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:40d7995935b8455a"}
21:53:59 [21:53:59.614]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.615]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a40b690e6698f56 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a40b690e6698f56"}
21:53:59 [21:53:59.618]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJQ3Eq5/1 with map size 1073741824
21:53:59 [21:53:59.621]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.645]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0a40b690e6698f56 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0a40b690e6698f56"}
21:53:59 [21:53:59.646]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0a40b690e6698f56 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0a40b690e6698f56"}
21:53:59 [21:53:59.663]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.664]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41ff32f72b23ea28 epochNumber=1
{"provingJobId":"1:BASE_PARITY:41ff32f72b23ea28"}
21:53:59 [21:53:59.665]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2sMfXY/1 with map size 1073741824
21:53:59 [21:53:59.668]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.691]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:41ff32f72b23ea28 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:41ff32f72b23ea28"}
21:53:59 [21:53:59.691]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:41ff32f72b23ea28 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:41ff32f72b23ea28"}
21:53:59 [21:53:59.693]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:41ff32f72b23ea28: Error: db error
21:53:59 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:41ff32f72b23ea28"}
21:53:59 [21:53:59.700]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.701]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa384c1756f64b06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa384c1756f64b06"}
21:53:59 [21:53:59.702]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJ3CYar/1 with map size 1073741824
21:53:59 [21:53:59.707]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.736]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aa384c1756f64b06 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:aa384c1756f64b06"}
21:53:59 [21:53:59.736]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:aa384c1756f64b06 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:aa384c1756f64b06"}
21:53:59 [21:53:59.755]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.756]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f4ef54658194f7c epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f4ef54658194f7c"}
21:53:59 [21:53:59.758]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUDNLmn/1 with map size 1073741824
21:53:59 [21:53:59.762]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.792]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2f4ef54658194f7c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2f4ef54658194f7c"}
21:53:59 [21:53:59.793]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2f4ef54658194f7c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2f4ef54658194f7c"}
21:53:59 [21:53:59.794]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:2f4ef54658194f7c jobErr=test error: Error: db error
21:53:59 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:2f4ef54658194f7c"}
21:53:59 [21:53:59.798]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.799]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:32245592970ba099 not found
{"provingJobId":"1:BASE_PARITY:32245592970ba099"}
21:53:59 [21:53:59.803]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.804]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1feb445a8cc377a8 err=test error
{"provingJoId":"1:BASE_PARITY:1feb445a8cc377a8"}
21:53:59 [21:53:59.810]
INFO:
prover-client:proving-broker Proving Broker started
21:53:59 [21:53:59.821]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ffde6248618ae0bc epochNumber=1
{"provingJobId":"1:BASE_PARITY:ffde6248618ae0bc"}
21:53:59 [21:53:59.822]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testclrOUu/1 with map size 1073741824
21:53:59 [21:53:59.824]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.849]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ffde6248618ae0bc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ffde6248618ae0bc"}
21:53:59 [21:53:59.850]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ffde6248618ae0bc type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ffde6248618ae0bc"}
21:53:59 [21:53:59.868]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:b0244c2989bc1060 epochNumber=2
{"provingJobId":"2:BASE_PARITY:b0244c2989bc1060"}
21:53:59 [21:53:59.869]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testclrOUu/2 with map size 1073741824
21:53:59 [21:53:59.873]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:59 [21:53:59.904]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:b0244c2989bc1060 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:b0244c2989bc1060"}
21:53:59 [21:53:59.904]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:b0244c2989bc1060 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:b0244c2989bc1060"}
21:53:59 [21:53:59.985]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:a05f139c3a813d93 epochNumber=3
{"provingJobId":"3:BASE_PARITY:a05f139c3a813d93"}
21:53:59 [21:53:59.988]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testclrOUu/3 with map size 1073741824
21:53:59 [21:53:59.991]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:54:00 [21:54:00.028]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:54:00 [21:54:00.029]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:54:00 [21:54:00.076]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:fe287e868ff53258 epochNumber=4
{"provingJobId":"4:BASE_PARITY:fe287e868ff53258"}
21:54:00 [21:54:00.078]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testclrOUu/4 with map size 1073741824
21:54:00 [21:54:00.083]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:54:00 [21:54:00.108]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:54:00 [21:54:00.109]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
21:54:00 [21:54:00.163]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:0e92ab515034759f epochNumber=5
{"provingJobId":"5:BASE_PARITY:0e92ab515034759f"}
21:54:00 [21:54:00.165]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testclrOUu/5 with map size 1073741824
21:54:00 [21:54:00.168]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:54:00 [21:54:00.212]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:54:00 [21:54:00.213]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
21:54:00
PASS src/proving_broker/proving_broker.test.ts (
13.669 s)
21:54:00 ProvingBroker
21:54:00 Producer API
21:54:00
✓ refuses stale jobs (41 ms)
21:54:00
✓ enqueues jobs (3 ms)
21:54:00
✓ ignores duplicate jobs (4 ms)
21:54:00
✓ reports correct status when enqueuing repeat jobs (4 ms)
21:54:00
✓ throws an error in case of duplicate job IDs (24 ms)
21:54:00
✓ returns not-found status for non-existing jobs (1 ms)
21:54:00
✓ cancels jobs in queue (3 ms)
21:54:00
✓ cancels jobs in-progress (2 ms)
21:54:00
✓ returns job result if successful (10 ms)
21:54:00
✓ returns job error if failed (3 ms)
21:54:00
✓ correctly returns job status for concurrent writes (3 ms)
21:54:00
✓ refuses stale jobs (46 ms)
21:54:00
✓ enqueues jobs (32 ms)
21:54:00
✓ ignores duplicate jobs (31 ms)
21:54:00
✓ reports correct status when enqueuing repeat jobs (35 ms)
21:54:00
✓ throws an error in case of duplicate job IDs (29 ms)
21:54:00
✓ returns not-found status for non-existing jobs (3 ms)
21:54:00
✓ cancels jobs in queue (34 ms)
21:54:00
✓ cancels jobs in-progress (41 ms)
21:54:00
✓ returns job result if successful (35 ms)
21:54:00
✓ returns job error if failed (41 ms)
21:54:00
✓ correctly returns job status for concurrent writes (44 ms)
21:54:00 Consumer API
21:54:00
✓ returns undefined if no jobs are available (1 ms)
21:54:00
✓ returns jobs in priority order (2 ms)
21:54:00
✓ returns undefined if no jobs are available for the given allowList (2 ms)
21:54:00
✓ returns a job if it is in the allowList (3 ms)
21:54:00
✓ returns the most important job if it is in the allowList (4 ms)
21:54:00
✓ returns any job if filter is empty (2 ms)
21:54:00
✓ returns a new job when reporting job success (3 ms)
21:54:00
✓ returns a new job when reporting permanent error (2 ms)
21:54:00
✓ returns a new job when reporting retry-able error (2 ms)
21:54:00
✓ returns a new job when reporting progress if current one is cancelled (5 ms)
21:54:00
✓ returns a new job if job is already in progress elsewhere (1057 ms)
21:54:00
✓ avoids sending the same job to a new agent after a restart (1006 ms)
21:54:00
✓ avoids sending a completed job to a new agent after a restart (1006 ms)
21:54:00
✓ tracks job result if in progress (6 ms)
21:54:00
✓ tracks job result even if job is in queue (7 ms)
21:54:00
✓ ignores reported job error if unknown job (1 ms)
21:54:00
✓ ignores job result if unknown job (1 ms)
21:54:00
✓ returns undefined if no jobs are available (3 ms)
21:54:00
✓ returns jobs in priority order (51 ms)
21:54:00
✓ returns undefined if no jobs are available for the given allowList (25 ms)
21:54:00
✓ returns a job if it is in the allowList (67 ms)
21:54:00
✓ returns the most important job if it is in the allowList (68 ms)
21:54:00
✓ returns any job if filter is empty (73 ms)
21:54:00
✓ returns a new job when reporting job success (47 ms)
21:54:00
✓ returns a new job when reporting permanent error (40 ms)
21:54:00
✓ returns a new job when reporting retry-able error (30 ms)
21:54:00
✓ returns a new job when reporting progress if current one is cancelled (38 ms)
21:54:00
✓ returns a new job if job is already in progress elsewhere (1109 ms)
21:54:00
✓ avoids sending the same job to a new agent after a restart (1055 ms)
21:54:00
✓ avoids sending a completed job to a new agent after a restart (1073 ms)
21:54:00
✓ tracks job result if in progress (65 ms)
21:54:00
✓ tracks job result even if job is in queue (60 ms)
21:54:00
✓ ignores reported job error if unknown job (5 ms)
21:54:00
✓ ignores job result if unknown job (4 ms)
21:54:00 Timeouts
21:54:00
✓ tracks in progress jobs (1 ms)
21:54:00
✓ re-enqueues jobs that time out (109 ms)
21:54:00
✓ cancel stale jobs that time out (237 ms)
21:54:00
✓ keeps the jobs in progress while it is alive (204 ms)
21:54:00
✓ tracks in progress jobs (27 ms)
21:54:00
✓ re-enqueues jobs that time out (137 ms)
21:54:00
✓ cancel stale jobs that time out (309 ms)
21:54:00
✓ keeps the jobs in progress while it is alive (251 ms)
21:54:00 Retries
21:54:00
✓ retries jobs (3 ms)
21:54:00
✓ retries up to a maximum number of times (2 ms)
21:54:00
✓ passing retry=false does not retry (1 ms)
21:54:00
✓ does not retry if job is stale (29 ms)
21:54:00
✓ retries jobs (29 ms)
21:54:00
✓ retries up to a maximum number of times (39 ms)
21:54:00
✓ passing retry=false does not retry (39 ms)
21:54:00
✓ does not retry if job is stale (82 ms)
21:54:00 Database management
21:54:00
✓ re-enqueues proof requests on start (4 ms)
21:54:00
✓ restores proof results on start (2 ms)
21:54:00
✓ only re-enqueues unfinished jobs (2 ms)
21:54:00
✓ saves job when enqueued (3 ms)
21:54:00
✓ does not retain job if database fails to save (3 ms)
21:54:00
✓ saves job result (2 ms)
21:54:00
✓ saves result even if database fails to save (3 ms)
21:54:00
✓ saves job error (2 ms)
21:54:00
✓ saves job error even if database fails to save (3 ms)
21:54:00
✓ does not save job result if job is unknown (2 ms)
21:54:00
✓ does not save job error if job is unknown (1 ms)
21:54:00
✓ cleans up old jobs periodically (266 ms)
21:54:00
✓ re-enqueues proof requests on start (57 ms)
21:54:00
✓ restores proof results on start (76 ms)
21:54:00
✓ only re-enqueues unfinished jobs (71 ms)
21:54:00
✓ saves job when enqueued (31 ms)
21:54:00
✓ does not retain job if database fails to save (6 ms)
21:54:00
✓ saves job result (48 ms)
21:54:00
✓ saves result even if database fails to save (36 ms)
21:54:00
✓ saves job error (54 ms)
21:54:00
✓ saves job error even if database fails to save (45 ms)
21:54:00
✓ does not save job result if job is unknown (4 ms)
21:54:00
✓ does not save job error if job is unknown (5 ms)
21:54:00
✓ cleans up old jobs periodically (436 ms)
21:54:00
21:54:00
Test Suites: 1 passed, 1 total
21:54:00
Tests: 96 passed, 96 total
21:54:00
Snapshots: 0 total
21:54:00
Time: 13.793 s
21:54:00
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
21:54:00
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?