Parent Log: http://ci.aztec-labs.com/6199c9f37bd6630e Command: 557c108a8ed544a3 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/4b0e4dc09fa55f0bc808fa0c7c3e95e8fa997069 Env: REF_NAME=gh-readonly-queue/next/pr-15116-433ba6218aed45169029991a493e4232b30b8ec6 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Tue Jun 17 20:30:34 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15116_amd64_x4-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 20:30:38 [20:30:38.604] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.612] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6f06df21f7eec2b2 epochNumber=42 {"provingJobId":"1:BASE_PARITY:6f06df21f7eec2b2"} 20:30:38 [20:30:38.615] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:281ccc1d97a43965 epochNumber=1 {"provingJobId":"1:BASE_PARITY:281ccc1d97a43965"} 20:30:38 [20:30:38.649] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.650] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:410f5c07e9b5bcb3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:410f5c07e9b5bcb3"} 20:30:38 [20:30:38.651] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:23931052b3dd4b48 epochNumber=1 {"provingJobId":"1:BASE_PARITY:23931052b3dd4b48"} 20:30:38 [20:30:38.653] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.654] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b563741788d90b84 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b563741788d90b84"} 20:30:38 [20:30:38.655] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b563741788d90b84 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b563741788d90b84"} 20:30:38 [20:30:38.658] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.659] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f3629281630660a6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f3629281630660a6"} 20:30:38 [20:30:38.660] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f3629281630660a6 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f3629281630660a6"} 20:30:38 [20:30:38.660] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f3629281630660a6 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f3629281630660a6"} 20:30:38 [20:30:38.661] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f3629281630660a6 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f3629281630660a6"} 20:30:38 [20:30:38.662] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.663] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d59c0a882f05d1f1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d59c0a882f05d1f1"} 20:30:38 [20:30:38.686] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.687] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.688] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c85d2d3070468ec2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c85d2d3070468ec2"} 20:30:38 [20:30:38.689] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c85d2d3070468ec2 {"provingJobId":"1:BASE_PARITY:c85d2d3070468ec2"} 20:30:38 [20:30:38.689] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c85d2d3070468ec2 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c85d2d3070468ec2"} 20:30:38 [20:30:38.689] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c85d2d3070468ec2 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:c85d2d3070468ec2"} 20:30:38 [20:30:38.691] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.691] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:14bdf60a2fbd78c1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:14bdf60a2fbd78c1"} 20:30:38 [20:30:38.692] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:14bdf60a2fbd78c1 {"provingJobId":"1:BASE_PARITY:14bdf60a2fbd78c1"} 20:30:38 [20:30:38.692] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:14bdf60a2fbd78c1 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:14bdf60a2fbd78c1"} 20:30:38 [20:30:38.694] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.694] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:901497396c81bd65 epochNumber=1 {"provingJobId":"1:BASE_PARITY:901497396c81bd65"} 20:30:38 [20:30:38.695] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:901497396c81bd65 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:901497396c81bd65"} 20:30:38 [20:30:38.695] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:901497396c81bd65 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:901497396c81bd65"} 20:30:38 [20:30:38.696] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.697] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ce2f99e8ae975481 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ce2f99e8ae975481"} 20:30:38 [20:30:38.697] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ce2f99e8ae975481 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ce2f99e8ae975481"} 20:30:38 [20:30:38.698] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ce2f99e8ae975481 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ce2f99e8ae975481"} 20:30:38 [20:30:38.699] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.699] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fb1ebc465618f9d8 epochNumber=0 {"provingJobId":"1:BASE_PARITY:fb1ebc465618f9d8"} 20:30:38 [20:30:38.699] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fb1ebc465618f9d8 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:fb1ebc465618f9d8"} 20:30:38 [20:30:38.700] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d985c6fe2d4107c3 epochNumber=0 {"provingJobId":"1:BASE_PARITY:d985c6fe2d4107c3"} 20:30:38 [20:30:38.700] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fb1ebc465618f9d8 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:fb1ebc465618f9d8"} 20:30:38 [20:30:38.700] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:726d08b16a8132e9 epochNumber=0 {"provingJobId":"1:BASE_PARITY:726d08b16a8132e9"} 20:30:38 [20:30:38.710] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.712] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.712] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4df0b7f6db112f9a epochNumber=2 {"provingJobId":"1:BASE_PARITY:4df0b7f6db112f9a"} 20:30:38 [20:30:38.713] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:04eade49233f1c16 epochNumber=1 {"provingJobId":"1:BASE_PARITY:04eade49233f1c16"} 20:30:38 [20:30:38.714] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.715] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4cce335af14d99fb epochNumber=1 {"provingJobId":"1:BASE_PARITY:4cce335af14d99fb"} 20:30:38 [20:30:38.716] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.717] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1042d97cce0ddba6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1042d97cce0ddba6"} 20:30:38 [20:30:38.717] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7a24a3b62f89dc1c epochNumber=1 {"provingJobId":"1:BASE_PARITY:7a24a3b62f89dc1c"} 20:30:38 [20:30:38.717] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9dd23121aa7bc653 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9dd23121aa7bc653"} 20:30:38 [20:30:38.718] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:355f5e6a65da823c epochNumber=1 {"provingJobId":"1:BASE_PARITY:355f5e6a65da823c"} 20:30:38 [20:30:38.719] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.720] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5826724a8930c61e epochNumber=1 {"provingJobId":"1:BASE_PARITY:5826724a8930c61e"} 20:30:38 [20:30:38.720] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:71fcfceb30688d0c epochNumber=1 {"provingJobId":"1:BASE_PARITY:71fcfceb30688d0c"} 20:30:38 [20:30:38.720] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f1eb52ae8c98bfa0 epochNumber=2 {"provingJobId":"1:BASE_PARITY:f1eb52ae8c98bfa0"} 20:30:38 [20:30:38.721] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:27b0b68c09bbd66f epochNumber=1 {"provingJobId":"1:BASE_PARITY:27b0b68c09bbd66f"} 20:30:38 [20:30:38.722] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.723] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6d2edfebaaf71407 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6d2edfebaaf71407"} 20:30:38 [20:30:38.723] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:06cca9782e66cf44 epochNumber=1 {"provingJobId":"1:BASE_PARITY:06cca9782e66cf44"} 20:30:38 [20:30:38.723] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f9c592d230c0817e epochNumber=2 {"provingJobId":"1:BASE_PARITY:f9c592d230c0817e"} 20:30:38 [20:30:38.724] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e5124da8afb3878e epochNumber=1 {"provingJobId":"1:BASE_PARITY:e5124da8afb3878e"} 20:30:38 [20:30:38.725] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.726] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:64e68268f02d4086 epochNumber=1 {"provingJobId":"1:BASE_PARITY:64e68268f02d4086"} 20:30:38 [20:30:38.726] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3aa94690c79bfd06 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3aa94690c79bfd06"} 20:30:38 [20:30:38.726] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:64e68268f02d4086 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:64e68268f02d4086"} 20:30:38 [20:30:38.728] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.728] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:139753af65fa9ea5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:139753af65fa9ea5"} 20:30:38 [20:30:38.729] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e85fc379d4692813 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e85fc379d4692813"} 20:30:38 [20:30:38.729] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:139753af65fa9ea5 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:139753af65fa9ea5"} 20:30:38 [20:30:38.730] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.731] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:83c1ea69e09e9c8b epochNumber=1 {"provingJobId":"1:BASE_PARITY:83c1ea69e09e9c8b"} 20:30:38 [20:30:38.732] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:84b15ed8dad193c5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:84b15ed8dad193c5"} 20:30:38 [20:30:38.732] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:83c1ea69e09e9c8b type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:83c1ea69e09e9c8b"} 20:30:38 [20:30:38.733] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.733] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:96367d5675cee0e7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:96367d5675cee0e7"} 20:30:38 [20:30:38.734] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:96367d5675cee0e7 {"provingJobId":"1:BASE_PARITY:96367d5675cee0e7"} 20:30:38 [20:30:38.734] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:96367d5675cee0e7 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:96367d5675cee0e7"} 20:30:38 [20:30:38.735] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b03cd51ac80ae1c8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b03cd51ac80ae1c8"} 20:30:38 [20:30:38.735] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:96367d5675cee0e7 has already been completed {"provingJobId":"1:BASE_PARITY:96367d5675cee0e7"} 20:30:38 [20:30:38.736] INFO: prover-client:proving-broker Proving Broker started 20:30:38 [20:30:38.737] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2d67c706607313ee epochNumber=1 {"provingJobId":"1:BASE_PARITY:2d67c706607313ee"} 20:30:38 [20:30:38.737] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7fbd43af12ed635d epochNumber=2 {"provingJobId":"1:BASE_PARITY:7fbd43af12ed635d"} 20:30:39 [20:30:39.791] INFO: prover-client:proving-broker Proving Broker started 20:30:39 [20:30:39.792] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2d67c706607313ee settled=false {"provingJobId":"1:BASE_PARITY:2d67c706607313ee","status":"pending"} 20:30:39 [20:30:39.792] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7fbd43af12ed635d settled=false {"provingJobId":"1:BASE_PARITY:7fbd43af12ed635d","status":"pending"} 20:30:39 [20:30:39.793] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:2d67c706607313ee type=BASE_PARITY startedAt=1750192238738 older agent has taken job {"provingJobId":"1:BASE_PARITY:2d67c706607313ee"} 20:30:39 [20:30:39.794] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2d67c706607313ee type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:2d67c706607313ee"} 20:30:39 [20:30:39.795] INFO: prover-client:proving-broker Proving Broker started 20:30:39 [20:30:39.796] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bf919c4bdab7c0bd epochNumber=1 {"provingJobId":"1:BASE_PARITY:bf919c4bdab7c0bd"} 20:30:39 [20:30:39.796] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6529a9420ab00f8c epochNumber=2 {"provingJobId":"1:BASE_PARITY:6529a9420ab00f8c"} 20:30:40 [20:30:40.797] INFO: prover-client:proving-broker Proving Broker started 20:30:40 [20:30:40.798] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bf919c4bdab7c0bd settled=false {"provingJobId":"1:BASE_PARITY:bf919c4bdab7c0bd","status":"pending"} 20:30:40 [20:30:40.798] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6529a9420ab00f8c settled=false {"provingJobId":"1:BASE_PARITY:6529a9420ab00f8c","status":"pending"} 20:30:40 [20:30:40.798] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bf919c4bdab7c0bd type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:bf919c4bdab7c0bd"} 20:30:40 [20:30:40.800] INFO: prover-client:proving-broker Proving Broker started 20:30:40 [20:30:40.800] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6686be81b3b01776 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6686be81b3b01776"} 20:30:40 [20:30:40.803] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4fc3efe4ab397419 epochNumber=2 {"provingJobId":"1:BASE_PARITY:4fc3efe4ab397419"} 20:30:41 [20:30:41.803] INFO: prover-client:proving-broker Proving Broker started 20:30:41 [20:30:41.804] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6686be81b3b01776 settled=false {"provingJobId":"1:BASE_PARITY:6686be81b3b01776","status":"pending"} 20:30:41 [20:30:41.804] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4fc3efe4ab397419 settled=false {"provingJobId":"1:BASE_PARITY:4fc3efe4ab397419","status":"pending"} 20:30:41 [20:30:41.805] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6686be81b3b01776 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:6686be81b3b01776"} 20:30:41 [20:30:41.805] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6686be81b3b01776 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6686be81b3b01776"} 20:30:41 [20:30:41.807] INFO: prover-client:proving-broker Proving Broker started 20:30:41 [20:30:41.808] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:822b6fc05ba2d779 epochNumber=1 {"provingJobId":"1:BASE_PARITY:822b6fc05ba2d779"} 20:30:41 [20:30:41.808] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fffbbf2ae3d698fe epochNumber=2 {"provingJobId":"1:BASE_PARITY:fffbbf2ae3d698fe"} 20:30:41 [20:30:41.809] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:822b6fc05ba2d779 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:822b6fc05ba2d779"} 20:30:41 [20:30:41.810] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fffbbf2ae3d698fe type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:fffbbf2ae3d698fe"} 20:30:41 [20:30:41.811] INFO: prover-client:proving-broker Proving Broker started 20:30:41 [20:30:41.811] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d55a76734a3436af epochNumber=1 {"provingJobId":"1:BASE_PARITY:d55a76734a3436af"} 20:30:41 [20:30:41.812] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3dd085ca06697eb0 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3dd085ca06697eb0"} 20:30:41 [20:30:41.812] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d55a76734a3436af type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d55a76734a3436af"} 20:30:41 [20:30:41.812] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d55a76734a3436af type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d55a76734a3436af"} 20:30:41 [20:30:41.812] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3dd085ca06697eb0 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3dd085ca06697eb0"} 20:30:41 [20:30:41.814] INFO: prover-client:proving-broker Proving Broker started 20:30:41 [20:30:41.814] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0fb81a4d51844e1d err=test error {"provingJoId":"1:BASE_PARITY:0fb81a4d51844e1d"} 20:30:41 [20:30:41.815] INFO: prover-client:proving-broker Proving Broker started 20:30:41 [20:30:41.816] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:14f140dc6f4f287f not found {"provingJobId":"1:BASE_PARITY:14f140dc6f4f287f"} 20:30:41 [20:30:41.817] INFO: prover-client:proving-broker Proving Broker started 20:30:41 [20:30:41.818] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1c339ae6dcc06dab epochNumber=1 {"provingJobId":"1:BASE_PARITY:1c339ae6dcc06dab"} 20:30:41 [20:30:41.819] INFO: prover-client:proving-broker Proving Broker started 20:30:41 [20:30:41.820] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8605f6ce04a89289 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8605f6ce04a89289"} 20:30:41 [20:30:41.921] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8605f6ce04a89289 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:8605f6ce04a89289"} 20:30:41 [20:30:41.929] INFO: prover-client:proving-broker Proving Broker started 20:30:41 [20:30:41.930] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:47173f27e5878efc epochNumber=1 {"provingJobId":"1:BASE_PARITY:47173f27e5878efc"} 20:30:42 [20:30:42.031] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:47173f27e5878efc timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:47173f27e5878efc"} 20:30:42 [20:30:42.038] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ad7471cd2f83d52c epochNumber=10 {"provingJobId":"1:BASE_PARITY:ad7471cd2f83d52c"} 20:30:42 [20:30:42.058] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:42 [20:30:42.167] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.168] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2bcf5a670d38810c epochNumber=1 {"provingJobId":"1:BASE_PARITY:2bcf5a670d38810c"} 20:30:42 [20:30:42.321] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2bcf5a670d38810c timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:2bcf5a670d38810c"} 20:30:42 [20:30:42.372] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.373] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:06e25369a5d1348a epochNumber=1 {"provingJobId":"1:BASE_PARITY:06e25369a5d1348a"} 20:30:42 [20:30:42.374] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:06e25369a5d1348a type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:06e25369a5d1348a"} 20:30:42 [20:30:42.376] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.376] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4ee567613e986241 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4ee567613e986241"} 20:30:42 [20:30:42.377] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4ee567613e986241 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:4ee567613e986241"} 20:30:42 [20:30:42.378] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4ee567613e986241 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:4ee567613e986241"} 20:30:42 [20:30:42.379] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.379] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cfc8979f81172750 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cfc8979f81172750"} 20:30:42 [20:30:42.380] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cfc8979f81172750 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:cfc8979f81172750"} 20:30:42 [20:30:42.381] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.381] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1635a1486b23aeaf epochNumber=1 {"provingJobId":"1:BASE_PARITY:1635a1486b23aeaf"} 20:30:42 [20:30:42.382] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1635a1486b23aeaf type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:1635a1486b23aeaf"} 20:30:42 [20:30:42.382] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a0543018469acd55 epochNumber=3 {"provingJobId":"1:BASE_PARITY:a0543018469acd55"} 20:30:42 [20:30:42.406] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:42 [20:30:42.410] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1635a1486b23aeaf err=test error {"provingJoId":"1:BASE_PARITY:1635a1486b23aeaf"} 20:30:42 [20:30:42.416] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.417] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:716ec61e4d48607c settled=false {"provingJobId":"1:BASE_PARITY:716ec61e4d48607c","status":"pending"} 20:30:42 [20:30:42.417] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:89105d20d838e518 settled=false {"provingJobId":"1:BASE_PARITY:89105d20d838e518","status":"pending"} 20:30:42 [20:30:42.420] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.420] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:436c769aa603758f settled=true {"provingJobId":"1:BASE_PARITY:436c769aa603758f","status":"fulfilled"} 20:30:42 [20:30:42.420] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:325b810ada767492 settled=true {"provingJobId":"2:BASE_PARITY:325b810ada767492","status":"fulfilled"} 20:30:42 [20:30:42.421] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.422] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:52532fc9471397e2 settled=true {"provingJobId":"1:BASE_PARITY:52532fc9471397e2","status":"fulfilled"} 20:30:42 [20:30:42.422] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:584849bb3132969d settled=false {"provingJobId":"1:BASE_PARITY:584849bb3132969d","status":"pending"} 20:30:42 [20:30:42.423] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.425] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:07d08960869c4347 epochNumber=1 {"provingJobId":"1:BASE_PARITY:07d08960869c4347"} 20:30:42 [20:30:42.428] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.428] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6224ed2f90f34e06 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6224ed2f90f34e06"} 20:30:42 [20:30:42.429] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:6224ed2f90f34e06: Error: db error: Error: db error 20:30:42 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:6224ed2f90f34e06"} 20:30:42 [20:30:42.431] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.431] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3f89d3a4940a8266 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3f89d3a4940a8266"} 20:30:42 [20:30:42.432] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3f89d3a4940a8266 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3f89d3a4940a8266"} 20:30:42 [20:30:42.432] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3f89d3a4940a8266 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3f89d3a4940a8266"} 20:30:42 [20:30:42.433] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.434] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b4d4f4f5002a308f epochNumber=1 {"provingJobId":"1:BASE_PARITY:b4d4f4f5002a308f"} 20:30:42 [20:30:42.434] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b4d4f4f5002a308f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b4d4f4f5002a308f"} 20:30:42 [20:30:42.434] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b4d4f4f5002a308f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b4d4f4f5002a308f"} 20:30:42 [20:30:42.435] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:b4d4f4f5002a308f: Error: db error 20:30:42 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:b4d4f4f5002a308f"} 20:30:42 [20:30:42.436] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.437] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:360eae27c1b01f35 epochNumber=1 {"provingJobId":"1:BASE_PARITY:360eae27c1b01f35"} 20:30:42 [20:30:42.437] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:360eae27c1b01f35 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:360eae27c1b01f35"} 20:30:42 [20:30:42.437] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:360eae27c1b01f35 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:360eae27c1b01f35"} 20:30:42 [20:30:42.438] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.439] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:679ab6bcf91efca3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:679ab6bcf91efca3"} 20:30:42 [20:30:42.439] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:679ab6bcf91efca3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:679ab6bcf91efca3"} 20:30:42 [20:30:42.439] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:679ab6bcf91efca3 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:679ab6bcf91efca3"} 20:30:42 [20:30:42.440] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:679ab6bcf91efca3 jobErr=test error: Error: db error 20:30:42 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:679ab6bcf91efca3"} 20:30:42 [20:30:42.440] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.441] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:22242c9c13004eb7 not found {"provingJobId":"1:BASE_PARITY:22242c9c13004eb7"} 20:30:42 [20:30:42.442] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.442] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:5744f9a3b1d2f2af err=test error {"provingJoId":"1:BASE_PARITY:5744f9a3b1d2f2af"} 20:30:42 [20:30:42.443] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.453] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e4cad110f1c471ae epochNumber=1 {"provingJobId":"1:BASE_PARITY:e4cad110f1c471ae"} 20:30:42 [20:30:42.454] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e4cad110f1c471ae type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e4cad110f1c471ae"} 20:30:42 [20:30:42.454] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e4cad110f1c471ae type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e4cad110f1c471ae"} 20:30:42 [20:30:42.465] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:15824a73db804354 epochNumber=2 {"provingJobId":"2:BASE_PARITY:15824a73db804354"} 20:30:42 [20:30:42.465] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:15824a73db804354 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:15824a73db804354"} 20:30:42 [20:30:42.465] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:15824a73db804354 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:15824a73db804354"} 20:30:42 [20:30:42.528] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:8e3ea0932980f247 epochNumber=3 {"provingJobId":"3:BASE_PARITY:8e3ea0932980f247"} 20:30:42 [20:30:42.544] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:42 [20:30:42.590] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:1d44984efaddccb9 epochNumber=4 {"provingJobId":"4:BASE_PARITY:1d44984efaddccb9"} 20:30:42 [20:30:42.596] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:42 [20:30:42.652] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:f68e976fc4219832 epochNumber=5 {"provingJobId":"5:BASE_PARITY:f68e976fc4219832"} 20:30:42 [20:30:42.674] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:42 [20:30:42.715] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.716] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c777ac3581ae6738 epochNumber=42 {"provingJobId":"1:BASE_PARITY:c777ac3581ae6738"} 20:30:42 [20:30:42.717] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testwDTHSk/42 with map size 1073741824 20:30:42 [20:30:42.724] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:42 [20:30:42.756] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:c12078bffea86891 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c12078bffea86891"} 20:30:42 [20:30:42.760] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.761] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1a0eb3d9c37e951b epochNumber=1 {"provingJobId":"1:BASE_PARITY:1a0eb3d9c37e951b"} 20:30:42 [20:30:42.762] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testc9KSF2/1 with map size 1073741824 20:30:42 [20:30:42.763] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:42 [20:30:42.787] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0b9547b362f2ce43 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0b9547b362f2ce43"} 20:30:42 [20:30:42.800] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.801] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ce893a431ca46b90 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ce893a431ca46b90"} 20:30:42 [20:30:42.801] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBSEHW3/1 with map size 1073741824 20:30:42 [20:30:42.806] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:42 [20:30:42.833] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ce893a431ca46b90 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:ce893a431ca46b90"} 20:30:42 [20:30:42.837] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.838] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2d4768dfd5201671 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2d4768dfd5201671"} 20:30:42 [20:30:42.838] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnWhF9q/1 with map size 1073741824 20:30:42 [20:30:42.840] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:42 [20:30:42.858] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2d4768dfd5201671 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:2d4768dfd5201671"} 20:30:42 [20:30:42.859] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2d4768dfd5201671 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2d4768dfd5201671"} 20:30:42 [20:30:42.864] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2d4768dfd5201671 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:2d4768dfd5201671"} 20:30:42 [20:30:42.868] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.869] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7f381b2fb9c07db4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7f381b2fb9c07db4"} 20:30:42 [20:30:42.869] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQrlFqG/1 with map size 1073741824 20:30:42 [20:30:42.871] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:42 [20:30:42.894] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.902] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.903] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:998ba1a00328ed45 epochNumber=1 {"provingJobId":"1:BASE_PARITY:998ba1a00328ed45"} 20:30:42 [20:30:42.904] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testR0w9Bi/1 with map size 1073741824 20:30:42 [20:30:42.909] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:42 [20:30:42.930] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:998ba1a00328ed45 {"provingJobId":"1:BASE_PARITY:998ba1a00328ed45"} 20:30:42 [20:30:42.930] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:998ba1a00328ed45 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:998ba1a00328ed45"} 20:30:42 [20:30:42.930] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:998ba1a00328ed45 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:998ba1a00328ed45"} 20:30:42 [20:30:42.940] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.940] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1b563ffe79ac2719 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1b563ffe79ac2719"} 20:30:42 [20:30:42.941] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8ggwKc/1 with map size 1073741824 20:30:42 [20:30:42.946] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:42 [20:30:42.965] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1b563ffe79ac2719 {"provingJobId":"1:BASE_PARITY:1b563ffe79ac2719"} 20:30:42 [20:30:42.966] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1b563ffe79ac2719 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:1b563ffe79ac2719"} 20:30:42 [20:30:42.976] INFO: prover-client:proving-broker Proving Broker started 20:30:42 [20:30:42.977] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d7f3cf03c9b6d3ed epochNumber=1 {"provingJobId":"1:BASE_PARITY:d7f3cf03c9b6d3ed"} 20:30:42 [20:30:42.979] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqeiLw4/1 with map size 1073741824 20:30:42 [20:30:42.988] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.010] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d7f3cf03c9b6d3ed type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d7f3cf03c9b6d3ed"} 20:30:43 [20:30:43.011] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d7f3cf03c9b6d3ed type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d7f3cf03c9b6d3ed"} 20:30:43 [20:30:43.019] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.020] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0cf1b362b7425076 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0cf1b362b7425076"} 20:30:43 [20:30:43.024] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPkj1O9/1 with map size 1073741824 20:30:43 [20:30:43.026] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.049] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0cf1b362b7425076 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0cf1b362b7425076"} 20:30:43 [20:30:43.049] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0cf1b362b7425076 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:0cf1b362b7425076"} 20:30:43 [20:30:43.060] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.060] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b758f428de5a3f0c epochNumber=0 {"provingJobId":"1:BASE_PARITY:b758f428de5a3f0c"} 20:30:43 [20:30:43.060] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test5oeOwA/0 with map size 1073741824 20:30:43 [20:30:43.062] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.082] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b758f428de5a3f0c epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b758f428de5a3f0c"} 20:30:43 [20:30:43.083] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e0bad71d1ae630b1 epochNumber=0 {"provingJobId":"1:BASE_PARITY:e0bad71d1ae630b1"} 20:30:43 [20:30:43.084] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b758f428de5a3f0c epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b758f428de5a3f0c"} 20:30:43 [20:30:43.084] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1eb7336e0e0b2d0b epochNumber=0 {"provingJobId":"1:BASE_PARITY:1eb7336e0e0b2d0b"} 20:30:43 [20:30:43.105] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.107] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.108] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9ec7f32e6a130ea5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9ec7f32e6a130ea5"} 20:30:43 [20:30:43.108] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQFYtIp/2 with map size 1073741824 20:30:43 [20:30:43.111] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.130] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ce027c4f346c2ed5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ce027c4f346c2ed5"} 20:30:43 [20:30:43.131] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQFYtIp/1 with map size 1073741824 20:30:43 [20:30:43.132] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.152] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.153] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3cc861f43fcd7dd5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3cc861f43fcd7dd5"} 20:30:43 [20:30:43.153] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwVtdMZ/1 with map size 1073741824 20:30:43 [20:30:43.159] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.179] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.180] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:522a70ce840f2751 epochNumber=1 {"provingJobId":"1:BASE_PARITY:522a70ce840f2751"} 20:30:43 [20:30:43.180] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8a4K6l/1 with map size 1073741824 20:30:43 [20:30:43.182] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.200] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f98561bc93d08989 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f98561bc93d08989"} 20:30:43 [20:30:43.207] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ccf7a7eed14bbdf7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:ccf7a7eed14bbdf7"} 20:30:43 [20:30:43.209] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8a4K6l/2 with map size 1073741824 20:30:43 [20:30:43.211] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.229] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1e0fd610f6eadfb2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1e0fd610f6eadfb2"} 20:30:43 [20:30:43.239] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.240] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6bfe964d56d1c35f epochNumber=1 {"provingJobId":"1:BASE_PARITY:6bfe964d56d1c35f"} 20:30:43 [20:30:43.240] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3lyjBv/1 with map size 1073741824 20:30:43 [20:30:43.242] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.259] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:349bead9e8220c64 epochNumber=1 {"provingJobId":"1:BASE_PARITY:349bead9e8220c64"} 20:30:43 [20:30:43.265] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e9b6d9417803270a epochNumber=2 {"provingJobId":"1:BASE_PARITY:e9b6d9417803270a"} 20:30:43 [20:30:43.266] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test3lyjBv/2 with map size 1073741824 20:30:43 [20:30:43.268] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.286] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b5ce3607de3ebba3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b5ce3607de3ebba3"} 20:30:43 [20:30:43.295] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.295] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:46c2793c2a3d0afb epochNumber=1 {"provingJobId":"1:BASE_PARITY:46c2793c2a3d0afb"} 20:30:43 [20:30:43.298] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOOSMVU/1 with map size 1073741824 20:30:43 [20:30:43.300] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.321] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d67db7459afd27eb epochNumber=1 {"provingJobId":"1:BASE_PARITY:d67db7459afd27eb"} 20:30:43 [20:30:43.327] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7a2d8f9ca707a2b5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:7a2d8f9ca707a2b5"} 20:30:43 [20:30:43.327] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOOSMVU/2 with map size 1073741824 20:30:43 [20:30:43.329] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.345] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:92eafdf150e9ee8d epochNumber=1 {"provingJobId":"1:BASE_PARITY:92eafdf150e9ee8d"} 20:30:43 [20:30:43.355] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.355] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6ab6dbb32605be7c epochNumber=1 {"provingJobId":"1:BASE_PARITY:6ab6dbb32605be7c"} 20:30:43 [20:30:43.356] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testc092eZ/1 with map size 1073741824 20:30:43 [20:30:43.358] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.374] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:37d45356cd763c02 epochNumber=1 {"provingJobId":"1:BASE_PARITY:37d45356cd763c02"} 20:30:43 [20:30:43.379] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6ab6dbb32605be7c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6ab6dbb32605be7c"} 20:30:43 [20:30:43.393] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.394] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:40a7e377cf094519 epochNumber=1 {"provingJobId":"1:BASE_PARITY:40a7e377cf094519"} 20:30:43 [20:30:43.395] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8uJ6yX/1 with map size 1073741824 20:30:43 [20:30:43.401] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.422] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:57b496af323f5097 epochNumber=1 {"provingJobId":"1:BASE_PARITY:57b496af323f5097"} 20:30:43 [20:30:43.428] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:40a7e377cf094519 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:40a7e377cf094519"} 20:30:43 [20:30:43.437] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.438] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:75ae6ac92ecc84b3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:75ae6ac92ecc84b3"} 20:30:43 [20:30:43.439] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvBaKmW/1 with map size 1073741824 20:30:43 [20:30:43.441] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.457] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:87d253af231140ee epochNumber=1 {"provingJobId":"1:BASE_PARITY:87d253af231140ee"} 20:30:43 [20:30:43.462] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:75ae6ac92ecc84b3 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:75ae6ac92ecc84b3"} 20:30:43 [20:30:43.467] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.468] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1da8294eaa6e7b72 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1da8294eaa6e7b72"} 20:30:43 [20:30:43.469] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFTmopA/1 with map size 1073741824 20:30:43 [20:30:43.471] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.487] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1da8294eaa6e7b72 {"provingJobId":"1:BASE_PARITY:1da8294eaa6e7b72"} 20:30:43 [20:30:43.488] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1da8294eaa6e7b72 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:1da8294eaa6e7b72"} 20:30:43 [20:30:43.494] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:582301fe22b571eb epochNumber=1 {"provingJobId":"1:BASE_PARITY:582301fe22b571eb"} 20:30:43 [20:30:43.501] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1da8294eaa6e7b72 has already been completed {"provingJobId":"1:BASE_PARITY:1da8294eaa6e7b72"} 20:30:43 [20:30:43.511] INFO: prover-client:proving-broker Proving Broker started 20:30:43 [20:30:43.512] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e333f6ceae9f76b6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e333f6ceae9f76b6"} 20:30:43 [20:30:43.513] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8sZh13/1 with map size 1073741824 20:30:43 [20:30:43.515] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:43 [20:30:43.531] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8bf9c4def9424653 epochNumber=2 {"provingJobId":"1:BASE_PARITY:8bf9c4def9424653"} 20:30:43 [20:30:43.533] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8sZh13/2 with map size 1073741824 20:30:43 [20:30:43.535] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:44 [20:30:44.605] INFO: prover-client:proving-broker Proving Broker started 20:30:44 [20:30:44.613] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e333f6ceae9f76b6 settled=false {"provingJobId":"1:BASE_PARITY:e333f6ceae9f76b6","status":"pending"} 20:30:44 [20:30:44.616] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8bf9c4def9424653 settled=false {"provingJobId":"1:BASE_PARITY:8bf9c4def9424653","status":"pending"} 20:30:44 [20:30:44.617] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:e333f6ceae9f76b6 type=BASE_PARITY startedAt=1750192243553 older agent has taken job {"provingJobId":"1:BASE_PARITY:e333f6ceae9f76b6"} 20:30:44 [20:30:44.617] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e333f6ceae9f76b6 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:e333f6ceae9f76b6"} 20:30:44 [20:30:44.623] INFO: prover-client:proving-broker Proving Broker started 20:30:44 [20:30:44.624] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e4970efeb525e7fd epochNumber=1 {"provingJobId":"1:BASE_PARITY:e4970efeb525e7fd"} 20:30:44 [20:30:44.627] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtrXtG5/1 with map size 1073741824 20:30:44 [20:30:44.632] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:44 [20:30:44.653] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:709895ba4a831921 epochNumber=2 {"provingJobId":"1:BASE_PARITY:709895ba4a831921"} 20:30:44 [20:30:44.655] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtrXtG5/2 with map size 1073741824 20:30:44 [20:30:44.658] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:45 [20:30:45.676] INFO: prover-client:proving-broker Proving Broker started 20:30:45 [20:30:45.678] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e4970efeb525e7fd settled=false {"provingJobId":"1:BASE_PARITY:e4970efeb525e7fd","status":"pending"} 20:30:45 [20:30:45.679] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:709895ba4a831921 settled=false {"provingJobId":"1:BASE_PARITY:709895ba4a831921","status":"pending"} 20:30:45 [20:30:45.680] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e4970efeb525e7fd type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:e4970efeb525e7fd"} 20:30:45 [20:30:45.684] INFO: prover-client:proving-broker Proving Broker started 20:30:45 [20:30:45.684] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6fc8fa47449f10bc epochNumber=1 {"provingJobId":"1:BASE_PARITY:6fc8fa47449f10bc"} 20:30:45 [20:30:45.685] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testY5BV7b/1 with map size 1073741824 20:30:45 [20:30:45.686] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:45 [20:30:45.702] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3f1b1fd81d6858e7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3f1b1fd81d6858e7"} 20:30:45 [20:30:45.703] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testY5BV7b/2 with map size 1073741824 20:30:45 [20:30:45.705] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:46 [20:30:46.722] INFO: prover-client:proving-broker Proving Broker started 20:30:46 [20:30:46.724] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6fc8fa47449f10bc settled=false {"provingJobId":"1:BASE_PARITY:6fc8fa47449f10bc","status":"pending"} 20:30:46 [20:30:46.726] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3f1b1fd81d6858e7 settled=false {"provingJobId":"1:BASE_PARITY:3f1b1fd81d6858e7","status":"pending"} 20:30:46 [20:30:46.727] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6fc8fa47449f10bc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:6fc8fa47449f10bc"} 20:30:46 [20:30:46.727] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6fc8fa47449f10bc type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6fc8fa47449f10bc"} 20:30:46 [20:30:46.736] INFO: prover-client:proving-broker Proving Broker started 20:30:46 [20:30:46.737] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e0e16936ac970d13 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e0e16936ac970d13"} 20:30:46 [20:30:46.737] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfvgngX/1 with map size 1073741824 20:30:46 [20:30:46.739] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:46 [20:30:46.755] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c3cd7e34e884cf1b epochNumber=2 {"provingJobId":"1:BASE_PARITY:c3cd7e34e884cf1b"} 20:30:46 [20:30:46.756] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfvgngX/2 with map size 1073741824 20:30:46 [20:30:46.758] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:46 [20:30:46.775] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e0e16936ac970d13 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e0e16936ac970d13"} 20:30:46 [20:30:46.781] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c3cd7e34e884cf1b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:c3cd7e34e884cf1b"} 20:30:46 [20:30:46.792] INFO: prover-client:proving-broker Proving Broker started 20:30:46 [20:30:46.793] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a404e052f24d6dcc epochNumber=1 {"provingJobId":"1:BASE_PARITY:a404e052f24d6dcc"} 20:30:46 [20:30:46.794] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjAM4YR/1 with map size 1073741824 20:30:46 [20:30:46.801] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:46 [20:30:46.824] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3036126740591783 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3036126740591783"} 20:30:46 [20:30:46.826] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjAM4YR/2 with map size 1073741824 20:30:46 [20:30:46.830] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:46 [20:30:46.846] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a404e052f24d6dcc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a404e052f24d6dcc"} 20:30:46 [20:30:46.847] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a404e052f24d6dcc type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:a404e052f24d6dcc"} 20:30:46 [20:30:46.852] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3036126740591783 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3036126740591783"} 20:30:46 [20:30:46.864] INFO: prover-client:proving-broker Proving Broker started 20:30:46 [20:30:46.866] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0597eec71261aa17 err=test error {"provingJoId":"1:BASE_PARITY:0597eec71261aa17"} 20:30:46 [20:30:46.870] INFO: prover-client:proving-broker Proving Broker started 20:30:46 [20:30:46.871] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:88d2bc3c30cca045 not found {"provingJobId":"1:BASE_PARITY:88d2bc3c30cca045"} 20:30:46 [20:30:46.876] INFO: prover-client:proving-broker Proving Broker started 20:30:46 [20:30:46.877] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5c9126940b443c92 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5c9126940b443c92"} 20:30:46 [20:30:46.878] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCbBmQh/1 with map size 1073741824 20:30:46 [20:30:46.881] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:46 [20:30:46.919] INFO: prover-client:proving-broker Proving Broker started 20:30:46 [20:30:46.920] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7c9fba44ceea0255 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7c9fba44ceea0255"} 20:30:46 [20:30:46.921] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGYmvtT/1 with map size 1073741824 20:30:46 [20:30:46.923] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.045] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7c9fba44ceea0255 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:7c9fba44ceea0255"} 20:30:47 [20:30:47.052] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.053] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b22d3205bf7fabe4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b22d3205bf7fabe4"} 20:30:47 [20:30:47.055] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDq138S/1 with map size 1073741824 20:30:47 [20:30:47.058] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.206] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b22d3205bf7fabe4 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:b22d3205bf7fabe4"} 20:30:47 [20:30:47.207] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab107479819efc84 epochNumber=10 {"provingJobId":"1:BASE_PARITY:ab107479819efc84"} 20:30:47 [20:30:47.208] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testDq138S/10 with map size 1073741824 20:30:47 [20:30:47.211] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.232] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:47 [20:30:47.233] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 20:30:47 [20:30:47.359] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.360] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aa1e59d3f44c7c4e epochNumber=1 {"provingJobId":"1:BASE_PARITY:aa1e59d3f44c7c4e"} 20:30:47 [20:30:47.360] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test94QBLS/1 with map size 1073741824 20:30:47 [20:30:47.363] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.539] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:aa1e59d3f44c7c4e timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:aa1e59d3f44c7c4e"} 20:30:47 [20:30:47.586] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.587] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4cacda3a9845fd75 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4cacda3a9845fd75"} 20:30:47 [20:30:47.588] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWVwMbj/1 with map size 1073741824 20:30:47 [20:30:47.590] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.618] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4cacda3a9845fd75 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:4cacda3a9845fd75"} 20:30:47 [20:30:47.625] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.626] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:71e9f9f7bd2d89c7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:71e9f9f7bd2d89c7"} 20:30:47 [20:30:47.627] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0CkK8h/1 with map size 1073741824 20:30:47 [20:30:47.629] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.646] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:71e9f9f7bd2d89c7 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:71e9f9f7bd2d89c7"} 20:30:47 [20:30:47.647] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:71e9f9f7bd2d89c7 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:71e9f9f7bd2d89c7"} 20:30:47 [20:30:47.658] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.659] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:123e2b27461c2d35 epochNumber=1 {"provingJobId":"1:BASE_PARITY:123e2b27461c2d35"} 20:30:47 [20:30:47.659] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3XLI2y/1 with map size 1073741824 20:30:47 [20:30:47.661] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.678] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:123e2b27461c2d35 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:123e2b27461c2d35"} 20:30:47 [20:30:47.688] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.688] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e262a07d8f9719bb epochNumber=1 {"provingJobId":"1:BASE_PARITY:e262a07d8f9719bb"} 20:30:47 [20:30:47.689] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfuhGOK/1 with map size 1073741824 20:30:47 [20:30:47.691] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.714] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e262a07d8f9719bb type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:e262a07d8f9719bb"} 20:30:47 [20:30:47.716] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e70d9924195f6d86 epochNumber=3 {"provingJobId":"1:BASE_PARITY:e70d9924195f6d86"} 20:30:47 [20:30:47.717] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testfuhGOK/3 with map size 1073741824 20:30:47 [20:30:47.725] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.767] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:47 [20:30:47.768] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 20:30:47 [20:30:47.774] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e262a07d8f9719bb err=test error {"provingJoId":"1:BASE_PARITY:e262a07d8f9719bb"} 20:30:47 [20:30:47.782] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTNrzvy/1 with map size 1073741824 20:30:47 [20:30:47.785] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.805] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTNrzvy/2 with map size 1073741824 20:30:47 [20:30:47.808] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.829] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.831] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:013f3f232641232c settled=false {"provingJobId":"1:BASE_PARITY:013f3f232641232c","status":"pending"} 20:30:47 [20:30:47.832] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4dfa5ace66de899f settled=false {"provingJobId":"1:BASE_PARITY:4dfa5ace66de899f","status":"pending"} 20:30:47 [20:30:47.837] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnSnC3K/1 with map size 1073741824 20:30:47 [20:30:47.841] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.860] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnSnC3K/2 with map size 1073741824 20:30:47 [20:30:47.865] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.898] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.900] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1bceb387f20fda08 settled=true {"provingJobId":"1:BASE_PARITY:1bceb387f20fda08","status":"fulfilled"} 20:30:47 [20:30:47.902] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:1480196f5d34d7b4 settled=true {"provingJobId":"2:BASE_PARITY:1480196f5d34d7b4","status":"fulfilled"} 20:30:47 [20:30:47.909] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaWuiVg/1 with map size 1073741824 20:30:47 [20:30:47.911] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.932] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testaWuiVg/2 with map size 1073741824 20:30:47 [20:30:47.934] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.952] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.953] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0e57cc82b7f6609b settled=true {"provingJobId":"1:BASE_PARITY:0e57cc82b7f6609b","status":"fulfilled"} 20:30:47 [20:30:47.956] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:caee50c822fde693 settled=false {"provingJobId":"1:BASE_PARITY:caee50c822fde693","status":"pending"} 20:30:47 [20:30:47.962] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.962] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2eb536105abd8207 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2eb536105abd8207"} 20:30:47 [20:30:47.965] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTkDURk/1 with map size 1073741824 20:30:47 [20:30:47.967] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:47 [20:30:47.994] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:47.995] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1fe6edc7fa2d686b epochNumber=1 {"provingJobId":"1:BASE_PARITY:1fe6edc7fa2d686b"} 20:30:47 [20:30:47.996] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:1fe6edc7fa2d686b: Error: db error: Error: db error 20:30:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:1fe6edc7fa2d686b"} 20:30:47 [20:30:47.999] INFO: prover-client:proving-broker Proving Broker started 20:30:47 [20:30:48.000] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cd854809fd46a050 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cd854809fd46a050"} 20:30:48 [20:30:48.001] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQvFFVQ/1 with map size 1073741824 20:30:48 [20:30:48.002] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:48 [20:30:48.022] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cd854809fd46a050 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cd854809fd46a050"} 20:30:48 [20:30:48.022] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cd854809fd46a050 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cd854809fd46a050"} 20:30:48 [20:30:48.035] INFO: prover-client:proving-broker Proving Broker started 20:30:48 [20:30:48.036] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4c905b9c31ecbc38 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4c905b9c31ecbc38"} 20:30:48 [20:30:48.039] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkxjUjB/1 with map size 1073741824 20:30:48 [20:30:48.043] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:48 [20:30:48.063] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4c905b9c31ecbc38 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4c905b9c31ecbc38"} 20:30:48 [20:30:48.064] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4c905b9c31ecbc38 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:4c905b9c31ecbc38"} 20:30:48 [20:30:48.065] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:4c905b9c31ecbc38: Error: db error 20:30:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:4c905b9c31ecbc38"} 20:30:48 [20:30:48.070] INFO: prover-client:proving-broker Proving Broker started 20:30:48 [20:30:48.071] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e5ad23b0ab338988 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e5ad23b0ab338988"} 20:30:48 [20:30:48.072] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwdfpjr/1 with map size 1073741824 20:30:48 [20:30:48.074] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:48 [20:30:48.092] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e5ad23b0ab338988 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e5ad23b0ab338988"} 20:30:48 [20:30:48.092] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e5ad23b0ab338988 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:e5ad23b0ab338988"} 20:30:48 [20:30:48.109] INFO: prover-client:proving-broker Proving Broker started 20:30:48 [20:30:48.110] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:df8fe1a687637552 epochNumber=1 {"provingJobId":"1:BASE_PARITY:df8fe1a687637552"} 20:30:48 [20:30:48.112] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7yCwqC/1 with map size 1073741824 20:30:48 [20:30:48.115] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:48 [20:30:48.138] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:df8fe1a687637552 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:df8fe1a687637552"} 20:30:48 [20:30:48.138] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:df8fe1a687637552 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:df8fe1a687637552"} 20:30:48 [20:30:48.139] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:df8fe1a687637552 jobErr=test error: Error: db error 20:30:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:df8fe1a687637552"} 20:30:48 [20:30:48.147] INFO: prover-client:proving-broker Proving Broker started 20:30:48 [20:30:48.147] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:39c6d59ae6835fb1 not found {"provingJobId":"1:BASE_PARITY:39c6d59ae6835fb1"} 20:30:48 [20:30:48.152] INFO: prover-client:proving-broker Proving Broker started 20:30:48 [20:30:48.153] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6e8eb38402fa85d6 err=test error {"provingJoId":"1:BASE_PARITY:6e8eb38402fa85d6"} 20:30:48 [20:30:48.158] INFO: prover-client:proving-broker Proving Broker started 20:30:48 [20:30:48.169] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b773af7f7adcef07 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b773af7f7adcef07"} 20:30:48 [20:30:48.170] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5c5VHj/1 with map size 1073741824 20:30:48 [20:30:48.173] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:48 [20:30:48.203] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b773af7f7adcef07 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b773af7f7adcef07"} 20:30:48 [20:30:48.204] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b773af7f7adcef07 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b773af7f7adcef07"} 20:30:48 [20:30:48.221] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:8adbc9282b89c84a epochNumber=2 {"provingJobId":"2:BASE_PARITY:8adbc9282b89c84a"} 20:30:48 [20:30:48.223] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5c5VHj/2 with map size 1073741824 20:30:48 [20:30:48.225] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:48 [20:30:48.252] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:8adbc9282b89c84a type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:8adbc9282b89c84a"} 20:30:48 [20:30:48.252] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:8adbc9282b89c84a type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:8adbc9282b89c84a"} 20:30:48 [20:30:48.324] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:4d6f2e32b03815de epochNumber=3 {"provingJobId":"3:BASE_PARITY:4d6f2e32b03815de"} 20:30:48 [20:30:48.325] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test5c5VHj/3 with map size 1073741824 20:30:48 [20:30:48.329] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:48 [20:30:48.364] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:48 [20:30:48.364] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 20:30:48 [20:30:48.407] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:37ab7a56b0ccafd3 epochNumber=4 {"provingJobId":"4:BASE_PARITY:37ab7a56b0ccafd3"} 20:30:48 [20:30:48.408] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-test5c5VHj/4 with map size 1073741824 20:30:48 [20:30:48.411] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:48 [20:30:48.445] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:48 [20:30:48.446] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 20:30:48 [20:30:48.496] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:5e86e680fd650494 epochNumber=5 {"provingJobId":"5:BASE_PARITY:5e86e680fd650494"} 20:30:48 [20:30:48.497] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-test5c5VHj/5 with map size 1073741824 20:30:48 [20:30:48.502] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 20:30:48 [20:30:48.548] INFO: prover-client:proving-broker Cleaned up jobs=1 20:30:48 [20:30:48.548] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 20:30:48 PASS src/proving_broker/proving_broker.test.ts (13.242 s) 20:30:48 ProvingBroker 20:30:48 Producer API 20:30:48 ✓ refuses stale jobs (47 ms) 20:30:48 ✓ enqueues jobs (5 ms) 20:30:48 ✓ ignores duplicate jobs (4 ms) 20:30:48 ✓ reports correct status when enqueuing repeat jobs (5 ms) 20:30:48 ✓ throws an error in case of duplicate job IDs (23 ms) 20:30:48 ✓ returns not-found status for non-existing jobs (2 ms) 20:30:48 ✓ cancels jobs in queue (3 ms) 20:30:48 ✓ cancels jobs in-progress (2 ms) 20:30:48 ✓ returns job result if successful (3 ms) 20:30:48 ✓ returns job error if failed (2 ms) 20:30:48 ✓ correctly returns job status for concurrent writes (11 ms) 20:30:48 ✓ refuses stale jobs (53 ms) 20:30:48 ✓ enqueues jobs (39 ms) 20:30:48 ✓ ignores duplicate jobs (37 ms) 20:30:48 ✓ reports correct status when enqueuing repeat jobs (30 ms) 20:30:48 ✓ throws an error in case of duplicate job IDs (26 ms) 20:30:48 ✓ returns not-found status for non-existing jobs (3 ms) 20:30:48 ✓ cancels jobs in queue (43 ms) 20:30:48 ✓ cancels jobs in-progress (35 ms) 20:30:48 ✓ returns job result if successful (43 ms) 20:30:48 ✓ returns job error if failed (41 ms) 20:30:48 ✓ correctly returns job status for concurrent writes (44 ms) 20:30:48 Consumer API 20:30:48 ✓ returns undefined if no jobs are available (2 ms) 20:30:48 ✓ returns jobs in priority order (3 ms) 20:30:48 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 20:30:48 ✓ returns a job if it is in the allowList (3 ms) 20:30:48 ✓ returns the most important job if it is in the allowList (2 ms) 20:30:48 ✓ returns any job if filter is empty (2 ms) 20:30:48 ✓ returns a new job when reporting job success (2 ms) 20:30:48 ✓ returns a new job when reporting permanent error (3 ms) 20:30:48 ✓ returns a new job when reporting retry-able error (2 ms) 20:30:48 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 20:30:48 ✓ returns a new job if job is already in progress elsewhere (1058 ms) 20:30:48 ✓ avoids sending the same job to a new agent after a restart (1005 ms) 20:30:48 ✓ avoids sending a completed job to a new agent after a restart (1007 ms) 20:30:48 ✓ tracks job result if in progress (3 ms) 20:30:48 ✓ tracks job result even if job is in queue (2 ms) 20:30:48 ✓ ignores reported job error if unknown job (2 ms) 20:30:48 ✓ ignores job result if unknown job (1 ms) 20:30:48 ✓ returns undefined if no jobs are available (2 ms) 20:30:48 ✓ returns jobs in priority order (44 ms) 20:30:48 ✓ returns undefined if no jobs are available for the given allowList (26 ms) 20:30:48 ✓ returns a job if it is in the allowList (60 ms) 20:30:48 ✓ returns the most important job if it is in the allowList (55 ms) 20:30:48 ✓ returns any job if filter is empty (59 ms) 20:30:48 ✓ returns a new job when reporting job success (34 ms) 20:30:48 ✓ returns a new job when reporting permanent error (48 ms) 20:30:48 ✓ returns a new job when reporting retry-able error (28 ms) 20:30:48 ✓ returns a new job when reporting progress if current one is cancelled (40 ms) 20:30:48 ✓ returns a new job if job is already in progress elsewhere (1113 ms) 20:30:48 ✓ avoids sending the same job to a new agent after a restart (1062 ms) 20:30:48 ✓ avoids sending a completed job to a new agent after a restart (1052 ms) 20:30:48 ✓ tracks job result if in progress (54 ms) 20:30:48 ✓ tracks job result even if job is in queue (72 ms) 20:30:48 ✓ ignores reported job error if unknown job (6 ms) 20:30:48 ✓ ignores job result if unknown job (5 ms) 20:30:48 Timeouts 20:30:48 ✓ tracks in progress jobs (2 ms) 20:30:48 ✓ re-enqueues jobs that time out (109 ms) 20:30:48 ✓ cancel stale jobs that time out (238 ms) 20:30:48 ✓ keeps the jobs in progress while it is alive (205 ms) 20:30:48 ✓ tracks in progress jobs (38 ms) 20:30:48 ✓ re-enqueues jobs that time out (136 ms) 20:30:48 ✓ cancel stale jobs that time out (307 ms) 20:30:48 ✓ keeps the jobs in progress while it is alive (228 ms) 20:30:48 Retries 20:30:48 ✓ retries jobs (3 ms) 20:30:48 ✓ retries up to a maximum number of times (3 ms) 20:30:48 ✓ passing retry=false does not retry (2 ms) 20:30:48 ✓ does not retry if job is stale (34 ms) 20:30:48 ✓ retries jobs (37 ms) 20:30:48 ✓ retries up to a maximum number of times (32 ms) 20:30:48 ✓ passing retry=false does not retry (31 ms) 20:30:48 ✓ does not retry if job is stale (90 ms) 20:30:48 Database management 20:30:48 ✓ re-enqueues proof requests on start (5 ms) 20:30:48 ✓ restores proof results on start (2 ms) 20:30:48 ✓ only re-enqueues unfinished jobs (1 ms) 20:30:48 ✓ saves job when enqueued (4 ms) 20:30:48 ✓ does not retain job if database fails to save (3 ms) 20:30:48 ✓ saves job result (3 ms) 20:30:48 ✓ saves result even if database fails to save (2 ms) 20:30:48 ✓ saves job error (2 ms) 20:30:48 ✓ saves job error even if database fails to save (2 ms) 20:30:48 ✓ does not save job result if job is unknown (1 ms) 20:30:48 ✓ does not save job error if job is unknown (1 ms) 20:30:48 ✓ cleans up old jobs periodically (263 ms) 20:30:48 ✓ re-enqueues proof requests on start (57 ms) 20:30:48 ✓ restores proof results on start (71 ms) 20:30:48 ✓ only re-enqueues unfinished jobs (53 ms) 20:30:48 ✓ saves job when enqueued (32 ms) 20:30:48 ✓ does not retain job if database fails to save (6 ms) 20:30:48 ✓ saves job result (35 ms) 20:30:48 ✓ saves result even if database fails to save (36 ms) 20:30:48 ✓ saves job error (37 ms) 20:30:48 ✓ saves job error even if database fails to save (38 ms) 20:30:48 ✓ does not save job result if job is unknown (5 ms) 20:30:48 ✓ does not save job error if job is unknown (5 ms) 20:30:48 ✓ cleans up old jobs periodically (426 ms) 20:30:48 20:30:48 Test Suites: 1 passed, 1 total 20:30:48 Tests: 96 passed, 96 total 20:30:48 Snapshots: 0 total 20:30:48 Time: 13.362 s 20:30:48 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 20:30:48 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?