Parent Log: http://ci.aztec-labs.com/3ca30566e9ce997a Command: 24e90b7895ff1ed6 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/8cc8a56892d36191eefb49254ebbe49ea99ef4b8 Env: REF_NAME=gh-readonly-queue/next/pr-14921-78d87cb88633aee537d022d6fd8e57ceb74a3a95 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Wed Jun 18 11:34:25 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14921_amd64_x1-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 11:34:29 [11:34:29.790] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.797] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ea68cdb851b2584 epochNumber=42 {"provingJobId":"1:BASE_PARITY:8ea68cdb851b2584"} 11:34:29 [11:34:29.801] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:e782e22cc95987b5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e782e22cc95987b5"} 11:34:29 [11:34:29.830] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.831] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a8545db8c37c3385 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a8545db8c37c3385"} 11:34:29 [11:34:29.833] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a2563842f4d05425 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a2563842f4d05425"} 11:34:29 [11:34:29.835] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.836] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:31c5cc41fff7619d epochNumber=1 {"provingJobId":"1:BASE_PARITY:31c5cc41fff7619d"} 11:34:29 [11:34:29.837] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:31c5cc41fff7619d epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:31c5cc41fff7619d"} 11:34:29 [11:34:29.840] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.842] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:305cdcf7c4a997df epochNumber=1 {"provingJobId":"1:BASE_PARITY:305cdcf7c4a997df"} 11:34:29 [11:34:29.844] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:305cdcf7c4a997df epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:305cdcf7c4a997df"} 11:34:29 [11:34:29.845] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:305cdcf7c4a997df type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:305cdcf7c4a997df"} 11:34:29 [11:34:29.847] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:305cdcf7c4a997df epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:305cdcf7c4a997df"} 11:34:29 [11:34:29.849] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.850] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:218d95f0a190d984 epochNumber=1 {"provingJobId":"1:BASE_PARITY:218d95f0a190d984"} 11:34:29 [11:34:29.878] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.881] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.882] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e56f0574d212f5f7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e56f0574d212f5f7"} 11:34:29 [11:34:29.882] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e56f0574d212f5f7 {"provingJobId":"1:BASE_PARITY:e56f0574d212f5f7"} 11:34:29 [11:34:29.883] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e56f0574d212f5f7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e56f0574d212f5f7"} 11:34:29 [11:34:29.883] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e56f0574d212f5f7 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:e56f0574d212f5f7"} 11:34:29 [11:34:29.884] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.885] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:301dd5f12673d7d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:301dd5f12673d7d6"} 11:34:29 [11:34:29.886] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:301dd5f12673d7d6 {"provingJobId":"1:BASE_PARITY:301dd5f12673d7d6"} 11:34:29 [11:34:29.886] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:301dd5f12673d7d6 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:301dd5f12673d7d6"} 11:34:29 [11:34:29.887] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.888] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4dd09ca98ada763d epochNumber=1 {"provingJobId":"1:BASE_PARITY:4dd09ca98ada763d"} 11:34:29 [11:34:29.888] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4dd09ca98ada763d type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4dd09ca98ada763d"} 11:34:29 [11:34:29.889] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4dd09ca98ada763d type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:4dd09ca98ada763d"} 11:34:29 [11:34:29.890] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.890] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6f96da3a9a1ba9a1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6f96da3a9a1ba9a1"} 11:34:29 [11:34:29.891] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6f96da3a9a1ba9a1 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:6f96da3a9a1ba9a1"} 11:34:29 [11:34:29.891] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6f96da3a9a1ba9a1 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:6f96da3a9a1ba9a1"} 11:34:29 [11:34:29.891] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.892] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7a581e43177bac95 epochNumber=0 {"provingJobId":"1:BASE_PARITY:7a581e43177bac95"} 11:34:29 [11:34:29.892] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7a581e43177bac95 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:7a581e43177bac95"} 11:34:29 [11:34:29.892] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6c8a09c775dd08ed epochNumber=0 {"provingJobId":"1:BASE_PARITY:6c8a09c775dd08ed"} 11:34:29 [11:34:29.893] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7a581e43177bac95 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:7a581e43177bac95"} 11:34:29 [11:34:29.893] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c9fa551ff2c6025f epochNumber=0 {"provingJobId":"1:BASE_PARITY:c9fa551ff2c6025f"} 11:34:29 [11:34:29.894] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.895] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.895] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5afb100951a5507f epochNumber=2 {"provingJobId":"1:BASE_PARITY:5afb100951a5507f"} 11:34:29 [11:34:29.895] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:106bdaa87b9dcaae epochNumber=1 {"provingJobId":"1:BASE_PARITY:106bdaa87b9dcaae"} 11:34:29 [11:34:29.905] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.906] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ade2453def0a7241 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ade2453def0a7241"} 11:34:29 [11:34:29.907] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.908] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1e8f90bf5be1dcee epochNumber=1 {"provingJobId":"1:BASE_PARITY:1e8f90bf5be1dcee"} 11:34:29 [11:34:29.908] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b62033f45d429d9a epochNumber=1 {"provingJobId":"1:BASE_PARITY:b62033f45d429d9a"} 11:34:29 [11:34:29.908] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1fb1b36f9fd0b69a epochNumber=2 {"provingJobId":"1:BASE_PARITY:1fb1b36f9fd0b69a"} 11:34:29 [11:34:29.909] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:caa47d463fc41611 epochNumber=1 {"provingJobId":"1:BASE_PARITY:caa47d463fc41611"} 11:34:29 [11:34:29.910] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.910] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:482efc3afe85a8d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:482efc3afe85a8d6"} 11:34:29 [11:34:29.911] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2eec48891f691cf8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2eec48891f691cf8"} 11:34:29 [11:34:29.911] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:720491d18416c91a epochNumber=2 {"provingJobId":"1:BASE_PARITY:720491d18416c91a"} 11:34:29 [11:34:29.911] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f8d30e77b3357872 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f8d30e77b3357872"} 11:34:29 [11:34:29.912] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.913] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5175c60435f1644f epochNumber=1 {"provingJobId":"1:BASE_PARITY:5175c60435f1644f"} 11:34:29 [11:34:29.913] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:86343b2ff6cd8742 epochNumber=1 {"provingJobId":"1:BASE_PARITY:86343b2ff6cd8742"} 11:34:29 [11:34:29.913] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7133d26218e8eec0 epochNumber=2 {"provingJobId":"1:BASE_PARITY:7133d26218e8eec0"} 11:34:29 [11:34:29.913] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:10aaf501086cdd88 epochNumber=1 {"provingJobId":"1:BASE_PARITY:10aaf501086cdd88"} 11:34:29 [11:34:29.915] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.915] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:634881b9642f4356 epochNumber=1 {"provingJobId":"1:BASE_PARITY:634881b9642f4356"} 11:34:29 [11:34:29.916] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:422c07fb5aced701 epochNumber=1 {"provingJobId":"1:BASE_PARITY:422c07fb5aced701"} 11:34:29 [11:34:29.916] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:634881b9642f4356 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:634881b9642f4356"} 11:34:29 [11:34:29.917] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.917] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5ed2bd8169b59df0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5ed2bd8169b59df0"} 11:34:29 [11:34:29.918] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:32e5ee6401ab2bec epochNumber=1 {"provingJobId":"1:BASE_PARITY:32e5ee6401ab2bec"} 11:34:29 [11:34:29.918] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5ed2bd8169b59df0 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:5ed2bd8169b59df0"} 11:34:29 [11:34:29.919] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.919] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6a46dc830d0665ba epochNumber=1 {"provingJobId":"1:BASE_PARITY:6a46dc830d0665ba"} 11:34:29 [11:34:29.920] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2cc0c7111ed6b8d0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2cc0c7111ed6b8d0"} 11:34:29 [11:34:29.920] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6a46dc830d0665ba type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:6a46dc830d0665ba"} 11:34:29 [11:34:29.921] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.921] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d1b9dda09639c262 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d1b9dda09639c262"} 11:34:29 [11:34:29.922] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d1b9dda09639c262 {"provingJobId":"1:BASE_PARITY:d1b9dda09639c262"} 11:34:29 [11:34:29.922] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d1b9dda09639c262 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:d1b9dda09639c262"} 11:34:29 [11:34:29.923] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:160299678fe9d47f epochNumber=1 {"provingJobId":"1:BASE_PARITY:160299678fe9d47f"} 11:34:29 [11:34:29.923] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d1b9dda09639c262 has already been completed {"provingJobId":"1:BASE_PARITY:d1b9dda09639c262"} 11:34:29 [11:34:29.924] INFO: prover-client:proving-broker Proving Broker started 11:34:29 [11:34:29.924] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:20fdd128c8735907 epochNumber=1 {"provingJobId":"1:BASE_PARITY:20fdd128c8735907"} 11:34:29 [11:34:29.925] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d329a11bdcddbfb6 epochNumber=2 {"provingJobId":"1:BASE_PARITY:d329a11bdcddbfb6"} 11:34:30 [11:34:30.977] INFO: prover-client:proving-broker Proving Broker started 11:34:30 [11:34:30.978] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:20fdd128c8735907 settled=false {"provingJobId":"1:BASE_PARITY:20fdd128c8735907","status":"pending"} 11:34:30 [11:34:30.979] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d329a11bdcddbfb6 settled=false {"provingJobId":"1:BASE_PARITY:d329a11bdcddbfb6","status":"pending"} 11:34:30 [11:34:30.980] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:20fdd128c8735907 type=BASE_PARITY startedAt=1750246469925 older agent has taken job {"provingJobId":"1:BASE_PARITY:20fdd128c8735907"} 11:34:30 [11:34:30.981] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:20fdd128c8735907 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:20fdd128c8735907"} 11:34:30 [11:34:30.982] INFO: prover-client:proving-broker Proving Broker started 11:34:30 [11:34:30.983] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5be5edcb439e6c81 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5be5edcb439e6c81"} 11:34:30 [11:34:30.984] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ded1459538e85d8a epochNumber=2 {"provingJobId":"1:BASE_PARITY:ded1459538e85d8a"} 11:34:31 [11:34:31.985] INFO: prover-client:proving-broker Proving Broker started 11:34:31 [11:34:31.986] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5be5edcb439e6c81 settled=false {"provingJobId":"1:BASE_PARITY:5be5edcb439e6c81","status":"pending"} 11:34:31 [11:34:31.986] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ded1459538e85d8a settled=false {"provingJobId":"1:BASE_PARITY:ded1459538e85d8a","status":"pending"} 11:34:31 [11:34:31.987] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5be5edcb439e6c81 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:5be5edcb439e6c81"} 11:34:31 [11:34:31.990] INFO: prover-client:proving-broker Proving Broker started 11:34:31 [11:34:31.991] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e6b09d22ce596f3c epochNumber=1 {"provingJobId":"1:BASE_PARITY:e6b09d22ce596f3c"} 11:34:31 [11:34:31.991] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:312b417690fd7b77 epochNumber=2 {"provingJobId":"1:BASE_PARITY:312b417690fd7b77"} 11:34:32 [11:34:32.992] INFO: prover-client:proving-broker Proving Broker started 11:34:32 [11:34:32.993] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e6b09d22ce596f3c settled=false {"provingJobId":"1:BASE_PARITY:e6b09d22ce596f3c","status":"pending"} 11:34:32 [11:34:32.993] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:312b417690fd7b77 settled=false {"provingJobId":"1:BASE_PARITY:312b417690fd7b77","status":"pending"} 11:34:32 [11:34:32.994] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e6b09d22ce596f3c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e6b09d22ce596f3c"} 11:34:32 [11:34:32.995] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e6b09d22ce596f3c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e6b09d22ce596f3c"} 11:34:32 [11:34:32.997] INFO: prover-client:proving-broker Proving Broker started 11:34:32 [11:34:32.997] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:58a3c6174d685a48 epochNumber=1 {"provingJobId":"1:BASE_PARITY:58a3c6174d685a48"} 11:34:32 [11:34:32.998] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:60f812296d5332af epochNumber=2 {"provingJobId":"1:BASE_PARITY:60f812296d5332af"} 11:34:32 [11:34:32.998] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:58a3c6174d685a48 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:58a3c6174d685a48"} 11:34:32 [11:34:32.999] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:60f812296d5332af type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:60f812296d5332af"} 11:34:33 [11:34:33.001] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.001] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:980b2b7aac4cc2ff epochNumber=1 {"provingJobId":"1:BASE_PARITY:980b2b7aac4cc2ff"} 11:34:33 [11:34:33.002] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4fc5bdbd72f12b4e epochNumber=2 {"provingJobId":"1:BASE_PARITY:4fc5bdbd72f12b4e"} 11:34:33 [11:34:33.002] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:980b2b7aac4cc2ff type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:980b2b7aac4cc2ff"} 11:34:33 [11:34:33.002] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:980b2b7aac4cc2ff type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:980b2b7aac4cc2ff"} 11:34:33 [11:34:33.003] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4fc5bdbd72f12b4e type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4fc5bdbd72f12b4e"} 11:34:33 [11:34:33.004] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.005] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:21a900b4c4ff71b1 err=test error {"provingJoId":"1:BASE_PARITY:21a900b4c4ff71b1"} 11:34:33 [11:34:33.006] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.007] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:dab794c46fe452b0 not found {"provingJobId":"1:BASE_PARITY:dab794c46fe452b0"} 11:34:33 [11:34:33.008] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.009] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0621b71f881a457e epochNumber=1 {"provingJobId":"1:BASE_PARITY:0621b71f881a457e"} 11:34:33 [11:34:33.011] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.011] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b737f42017f25708 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b737f42017f25708"} 11:34:33 [11:34:33.113] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b737f42017f25708 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:b737f42017f25708"} 11:34:33 [11:34:33.120] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.121] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7ebcc04179ce1d44 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7ebcc04179ce1d44"} 11:34:33 [11:34:33.223] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7ebcc04179ce1d44 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:7ebcc04179ce1d44"} 11:34:33 [11:34:33.225] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:319dc5b9ee8021ae epochNumber=10 {"provingJobId":"1:BASE_PARITY:319dc5b9ee8021ae"} 11:34:33 [11:34:33.250] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:33 [11:34:33.354] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.355] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:62aa388b701f6e73 epochNumber=1 {"provingJobId":"1:BASE_PARITY:62aa388b701f6e73"} 11:34:33 [11:34:33.508] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:62aa388b701f6e73 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:62aa388b701f6e73"} 11:34:33 [11:34:33.558] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.559] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:afcee84fea65617f epochNumber=1 {"provingJobId":"1:BASE_PARITY:afcee84fea65617f"} 11:34:33 [11:34:33.561] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:afcee84fea65617f type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:afcee84fea65617f"} 11:34:33 [11:34:33.562] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.563] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:292305307ce15544 epochNumber=1 {"provingJobId":"1:BASE_PARITY:292305307ce15544"} 11:34:33 [11:34:33.563] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:292305307ce15544 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:292305307ce15544"} 11:34:33 [11:34:33.564] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:292305307ce15544 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:292305307ce15544"} 11:34:33 [11:34:33.565] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.565] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b13b2d2c4d571028 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b13b2d2c4d571028"} 11:34:33 [11:34:33.566] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b13b2d2c4d571028 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b13b2d2c4d571028"} 11:34:33 [11:34:33.568] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.568] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ebee536de7923cab epochNumber=1 {"provingJobId":"1:BASE_PARITY:ebee536de7923cab"} 11:34:33 [11:34:33.569] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ebee536de7923cab type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:ebee536de7923cab"} 11:34:33 [11:34:33.569] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e4b9d31887b31a19 epochNumber=3 {"provingJobId":"1:BASE_PARITY:e4b9d31887b31a19"} 11:34:33 [11:34:33.593] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:33 [11:34:33.595] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ebee536de7923cab err=test error {"provingJoId":"1:BASE_PARITY:ebee536de7923cab"} 11:34:33 [11:34:33.597] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.598] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1651e6f1abe1bcff settled=false {"provingJobId":"1:BASE_PARITY:1651e6f1abe1bcff","status":"pending"} 11:34:33 [11:34:33.598] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:074ec8036db1afb0 settled=false {"provingJobId":"1:BASE_PARITY:074ec8036db1afb0","status":"pending"} 11:34:33 [11:34:33.601] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.601] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b6f182b1acd2d40e settled=true {"provingJobId":"1:BASE_PARITY:b6f182b1acd2d40e","status":"fulfilled"} 11:34:33 [11:34:33.602] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:77b2317a34751139 settled=true {"provingJobId":"2:BASE_PARITY:77b2317a34751139","status":"fulfilled"} 11:34:33 [11:34:33.603] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.604] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0ec21a14a7026698 settled=true {"provingJobId":"1:BASE_PARITY:0ec21a14a7026698","status":"fulfilled"} 11:34:33 [11:34:33.604] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3edaa7976683cbf3 settled=false {"provingJobId":"1:BASE_PARITY:3edaa7976683cbf3","status":"pending"} 11:34:33 [11:34:33.606] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.607] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c47ecbd2bdad8b86 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c47ecbd2bdad8b86"} 11:34:33 [11:34:33.608] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.609] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7365fd25ec8b4877 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7365fd25ec8b4877"} 11:34:33 [11:34:33.609] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:7365fd25ec8b4877: Error: db error: Error: db error 11:34:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:7365fd25ec8b4877"} 11:34:33 [11:34:33.611] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.611] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b2d4c7866daf4ae0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b2d4c7866daf4ae0"} 11:34:33 [11:34:33.611] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b2d4c7866daf4ae0 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b2d4c7866daf4ae0"} 11:34:33 [11:34:33.611] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b2d4c7866daf4ae0 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b2d4c7866daf4ae0"} 11:34:33 [11:34:33.612] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.612] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2ba9c2ff8e985492 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2ba9c2ff8e985492"} 11:34:33 [11:34:33.613] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2ba9c2ff8e985492 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2ba9c2ff8e985492"} 11:34:33 [11:34:33.613] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2ba9c2ff8e985492 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2ba9c2ff8e985492"} 11:34:33 [11:34:33.614] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:2ba9c2ff8e985492: Error: db error 11:34:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:2ba9c2ff8e985492"} 11:34:33 [11:34:33.615] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.615] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:06b82a05c6dcf04a epochNumber=1 {"provingJobId":"1:BASE_PARITY:06b82a05c6dcf04a"} 11:34:33 [11:34:33.616] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:06b82a05c6dcf04a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:06b82a05c6dcf04a"} 11:34:33 [11:34:33.616] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:06b82a05c6dcf04a type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:06b82a05c6dcf04a"} 11:34:33 [11:34:33.617] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.618] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d211b3c1f2c6c440 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d211b3c1f2c6c440"} 11:34:33 [11:34:33.618] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d211b3c1f2c6c440 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d211b3c1f2c6c440"} 11:34:33 [11:34:33.619] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d211b3c1f2c6c440 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:d211b3c1f2c6c440"} 11:34:33 [11:34:33.619] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:d211b3c1f2c6c440 jobErr=test error: Error: db error 11:34:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:d211b3c1f2c6c440"} 11:34:33 [11:34:33.626] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.626] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:15ef3cfea3933e9f not found {"provingJobId":"1:BASE_PARITY:15ef3cfea3933e9f"} 11:34:33 [11:34:33.628] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.629] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bf4d5b02b2144c71 err=test error {"provingJoId":"1:BASE_PARITY:bf4d5b02b2144c71"} 11:34:33 [11:34:33.633] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.649] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4d4638bd71b5814f epochNumber=1 {"provingJobId":"1:BASE_PARITY:4d4638bd71b5814f"} 11:34:33 [11:34:33.650] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4d4638bd71b5814f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4d4638bd71b5814f"} 11:34:33 [11:34:33.651] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4d4638bd71b5814f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:4d4638bd71b5814f"} 11:34:33 [11:34:33.662] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:003c941e3451a0e7 epochNumber=2 {"provingJobId":"2:BASE_PARITY:003c941e3451a0e7"} 11:34:33 [11:34:33.662] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:003c941e3451a0e7 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:003c941e3451a0e7"} 11:34:33 [11:34:33.663] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:003c941e3451a0e7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:003c941e3451a0e7"} 11:34:33 [11:34:33.724] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:0586aec7dc39f9fa epochNumber=3 {"provingJobId":"3:BASE_PARITY:0586aec7dc39f9fa"} 11:34:33 [11:34:33.740] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:33 [11:34:33.785] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:0d8de1a966787dff epochNumber=4 {"provingJobId":"4:BASE_PARITY:0d8de1a966787dff"} 11:34:33 [11:34:33.795] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:33 [11:34:33.850] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:ec34c2472f64c897 epochNumber=5 {"provingJobId":"5:BASE_PARITY:ec34c2472f64c897"} 11:34:33 [11:34:33.874] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:33 [11:34:33.905] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.906] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e2c1bdfa58a0913d epochNumber=42 {"provingJobId":"1:BASE_PARITY:e2c1bdfa58a0913d"} 11:34:33 [11:34:33.907] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testQkVRSZ/42 with map size 1073741824 11:34:33 [11:34:33.912] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:33 [11:34:33.946] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:3474c3d6381b7513 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3474c3d6381b7513"} 11:34:33 [11:34:33.952] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.953] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:25ff9e21c43396a8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:25ff9e21c43396a8"} 11:34:33 [11:34:33.954] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9acf5Y/1 with map size 1073741824 11:34:33 [11:34:33.956] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:33 [11:34:33.975] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7a382ff41b1ad203 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7a382ff41b1ad203"} 11:34:33 [11:34:33.990] INFO: prover-client:proving-broker Proving Broker started 11:34:33 [11:34:33.991] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d1a9ecd1ebc273c6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d1a9ecd1ebc273c6"} 11:34:33 [11:34:33.993] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPSMWWv/1 with map size 1073741824 11:34:33 [11:34:33.997] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.019] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d1a9ecd1ebc273c6 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:d1a9ecd1ebc273c6"} 11:34:34 [11:34:34.023] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.023] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3e591187c4a7f816 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3e591187c4a7f816"} 11:34:34 [11:34:34.024] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaFEVqY/1 with map size 1073741824 11:34:34 [11:34:34.027] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.051] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3e591187c4a7f816 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:3e591187c4a7f816"} 11:34:34 [11:34:34.052] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3e591187c4a7f816 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3e591187c4a7f816"} 11:34:34 [11:34:34.058] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3e591187c4a7f816 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:3e591187c4a7f816"} 11:34:34 [11:34:34.062] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.063] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:46c95471194592c0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:46c95471194592c0"} 11:34:34 [11:34:34.064] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjTIwP1/1 with map size 1073741824 11:34:34 [11:34:34.066] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.089] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.091] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.092] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:240c23137110967c epochNumber=1 {"provingJobId":"1:BASE_PARITY:240c23137110967c"} 11:34:34 [11:34:34.093] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZWvGQp/1 with map size 1073741824 11:34:34 [11:34:34.096] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.116] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:240c23137110967c {"provingJobId":"1:BASE_PARITY:240c23137110967c"} 11:34:34 [11:34:34.116] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:240c23137110967c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:240c23137110967c"} 11:34:34 [11:34:34.117] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:240c23137110967c type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:240c23137110967c"} 11:34:34 [11:34:34.126] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.126] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c4022b8b2a931f8c epochNumber=1 {"provingJobId":"1:BASE_PARITY:c4022b8b2a931f8c"} 11:34:34 [11:34:34.127] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfsAq7Z/1 with map size 1073741824 11:34:34 [11:34:34.137] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.164] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c4022b8b2a931f8c {"provingJobId":"1:BASE_PARITY:c4022b8b2a931f8c"} 11:34:34 [11:34:34.165] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c4022b8b2a931f8c type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:c4022b8b2a931f8c"} 11:34:34 [11:34:34.175] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.176] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:06a876e0baefad8f epochNumber=1 {"provingJobId":"1:BASE_PARITY:06a876e0baefad8f"} 11:34:34 [11:34:34.177] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKRGTO2/1 with map size 1073741824 11:34:34 [11:34:34.179] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.206] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:06a876e0baefad8f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:06a876e0baefad8f"} 11:34:34 [11:34:34.206] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:06a876e0baefad8f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:06a876e0baefad8f"} 11:34:34 [11:34:34.217] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.217] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b016799971bf1894 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b016799971bf1894"} 11:34:34 [11:34:34.218] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdnxQfB/1 with map size 1073741824 11:34:34 [11:34:34.220] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.245] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b016799971bf1894 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b016799971bf1894"} 11:34:34 [11:34:34.246] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b016799971bf1894 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b016799971bf1894"} 11:34:34 [11:34:34.254] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.255] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c6c9ab195e7160ef epochNumber=0 {"provingJobId":"1:BASE_PARITY:c6c9ab195e7160ef"} 11:34:34 [11:34:34.255] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testTuwiwF/0 with map size 1073741824 11:34:34 [11:34:34.257] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.275] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c6c9ab195e7160ef epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:c6c9ab195e7160ef"} 11:34:34 [11:34:34.276] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e8a0070042eb09ab epochNumber=0 {"provingJobId":"1:BASE_PARITY:e8a0070042eb09ab"} 11:34:34 [11:34:34.276] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c6c9ab195e7160ef epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:c6c9ab195e7160ef"} 11:34:34 [11:34:34.276] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:28faaa9e3e5c23df epochNumber=0 {"provingJobId":"1:BASE_PARITY:28faaa9e3e5c23df"} 11:34:34 [11:34:34.289] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.291] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.291] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5c1d5fc45c3092a1 epochNumber=2 {"provingJobId":"1:BASE_PARITY:5c1d5fc45c3092a1"} 11:34:34 [11:34:34.292] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testRQRqsS/2 with map size 1073741824 11:34:34 [11:34:34.293] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.312] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ee69192b0d516d46 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ee69192b0d516d46"} 11:34:34 [11:34:34.315] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRQRqsS/1 with map size 1073741824 11:34:34 [11:34:34.317] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.342] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.343] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:30c879e3ba930b7a epochNumber=1 {"provingJobId":"1:BASE_PARITY:30c879e3ba930b7a"} 11:34:34 [11:34:34.343] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testG8izHu/1 with map size 1073741824 11:34:34 [11:34:34.349] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.371] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.372] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:754798563ab7487b epochNumber=1 {"provingJobId":"1:BASE_PARITY:754798563ab7487b"} 11:34:34 [11:34:34.373] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvlUAlD/1 with map size 1073741824 11:34:34 [11:34:34.374] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.392] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7ec40c4a27f6b0d8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7ec40c4a27f6b0d8"} 11:34:34 [11:34:34.398] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d7c0817592efe3e7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:d7c0817592efe3e7"} 11:34:34 [11:34:34.399] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvlUAlD/2 with map size 1073741824 11:34:34 [11:34:34.401] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.422] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:860d9c8d69ce21a8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:860d9c8d69ce21a8"} 11:34:34 [11:34:34.435] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.436] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3cc5c9d6cd7192c2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3cc5c9d6cd7192c2"} 11:34:34 [11:34:34.436] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQHt8k0/1 with map size 1073741824 11:34:34 [11:34:34.438] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.463] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:802517eff3679c88 epochNumber=1 {"provingJobId":"1:BASE_PARITY:802517eff3679c88"} 11:34:34 [11:34:34.469] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:13905be87c8a6a36 epochNumber=2 {"provingJobId":"1:BASE_PARITY:13905be87c8a6a36"} 11:34:34 [11:34:34.470] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQHt8k0/2 with map size 1073741824 11:34:34 [11:34:34.472] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.488] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:11be7412beb38dc4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:11be7412beb38dc4"} 11:34:34 [11:34:34.500] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.501] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:acaa94174847645e epochNumber=1 {"provingJobId":"1:BASE_PARITY:acaa94174847645e"} 11:34:34 [11:34:34.501] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2jfoBs/1 with map size 1073741824 11:34:34 [11:34:34.503] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.519] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:960c36dcbeccc57b epochNumber=1 {"provingJobId":"1:BASE_PARITY:960c36dcbeccc57b"} 11:34:34 [11:34:34.526] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:86e03444235941e3 epochNumber=2 {"provingJobId":"1:BASE_PARITY:86e03444235941e3"} 11:34:34 [11:34:34.527] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2jfoBs/2 with map size 1073741824 11:34:34 [11:34:34.531] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.555] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6b1772e345acbd9d epochNumber=1 {"provingJobId":"1:BASE_PARITY:6b1772e345acbd9d"} 11:34:34 [11:34:34.567] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.568] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f5302e283130d993 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f5302e283130d993"} 11:34:34 [11:34:34.569] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testplhui8/1 with map size 1073741824 11:34:34 [11:34:34.571] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.590] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:424fb42c79a8af40 epochNumber=1 {"provingJobId":"1:BASE_PARITY:424fb42c79a8af40"} 11:34:34 [11:34:34.597] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f5302e283130d993 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f5302e283130d993"} 11:34:34 [11:34:34.608] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.610] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8a0a7c036dac7a9b epochNumber=1 {"provingJobId":"1:BASE_PARITY:8a0a7c036dac7a9b"} 11:34:34 [11:34:34.610] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSJxmqW/1 with map size 1073741824 11:34:34 [11:34:34.613] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.633] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d08c3a8c43c16044 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d08c3a8c43c16044"} 11:34:34 [11:34:34.640] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8a0a7c036dac7a9b type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:8a0a7c036dac7a9b"} 11:34:34 [11:34:34.655] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.656] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:501a964d55e41d0a epochNumber=1 {"provingJobId":"1:BASE_PARITY:501a964d55e41d0a"} 11:34:34 [11:34:34.657] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzshvpS/1 with map size 1073741824 11:34:34 [11:34:34.659] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.676] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1aea663bcd8befde epochNumber=1 {"provingJobId":"1:BASE_PARITY:1aea663bcd8befde"} 11:34:34 [11:34:34.682] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:501a964d55e41d0a type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:501a964d55e41d0a"} 11:34:34 [11:34:34.686] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.686] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:abf12a51b1cd61e5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:abf12a51b1cd61e5"} 11:34:34 [11:34:34.687] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0xbk9I/1 with map size 1073741824 11:34:34 [11:34:34.688] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.704] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:abf12a51b1cd61e5 {"provingJobId":"1:BASE_PARITY:abf12a51b1cd61e5"} 11:34:34 [11:34:34.704] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:abf12a51b1cd61e5 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:abf12a51b1cd61e5"} 11:34:34 [11:34:34.710] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fdcd28591b63c14a epochNumber=1 {"provingJobId":"1:BASE_PARITY:fdcd28591b63c14a"} 11:34:34 [11:34:34.715] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:abf12a51b1cd61e5 has already been completed {"provingJobId":"1:BASE_PARITY:abf12a51b1cd61e5"} 11:34:34 [11:34:34.718] INFO: prover-client:proving-broker Proving Broker started 11:34:34 [11:34:34.719] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:83767d2c26cbe823 epochNumber=1 {"provingJobId":"1:BASE_PARITY:83767d2c26cbe823"} 11:34:34 [11:34:34.719] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAn7f1d/1 with map size 1073741824 11:34:34 [11:34:34.721] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:34 [11:34:34.739] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:82e89e826f45f8d8 epochNumber=2 {"provingJobId":"1:BASE_PARITY:82e89e826f45f8d8"} 11:34:34 [11:34:34.740] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAn7f1d/2 with map size 1073741824 11:34:34 [11:34:34.743] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:35 [11:34:35.810] INFO: prover-client:proving-broker Proving Broker started 11:34:35 [11:34:35.817] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:83767d2c26cbe823 settled=false {"provingJobId":"1:BASE_PARITY:83767d2c26cbe823","status":"pending"} 11:34:35 [11:34:35.819] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:82e89e826f45f8d8 settled=false {"provingJobId":"1:BASE_PARITY:82e89e826f45f8d8","status":"pending"} 11:34:35 [11:34:35.820] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:83767d2c26cbe823 type=BASE_PARITY startedAt=1750246474758 older agent has taken job {"provingJobId":"1:BASE_PARITY:83767d2c26cbe823"} 11:34:35 [11:34:35.821] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:83767d2c26cbe823 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:83767d2c26cbe823"} 11:34:35 [11:34:35.825] INFO: prover-client:proving-broker Proving Broker started 11:34:35 [11:34:35.826] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5baee96b4f65736b epochNumber=1 {"provingJobId":"1:BASE_PARITY:5baee96b4f65736b"} 11:34:35 [11:34:35.826] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAMq7NW/1 with map size 1073741824 11:34:35 [11:34:35.829] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:35 [11:34:35.846] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6944575d537adbe6 epochNumber=2 {"provingJobId":"1:BASE_PARITY:6944575d537adbe6"} 11:34:35 [11:34:35.847] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAMq7NW/2 with map size 1073741824 11:34:35 [11:34:35.849] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:36 [11:34:36.867] INFO: prover-client:proving-broker Proving Broker started 11:34:36 [11:34:36.870] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5baee96b4f65736b settled=false {"provingJobId":"1:BASE_PARITY:5baee96b4f65736b","status":"pending"} 11:34:36 [11:34:36.872] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6944575d537adbe6 settled=false {"provingJobId":"1:BASE_PARITY:6944575d537adbe6","status":"pending"} 11:34:36 [11:34:36.873] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5baee96b4f65736b type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:5baee96b4f65736b"} 11:34:36 [11:34:36.876] INFO: prover-client:proving-broker Proving Broker started 11:34:36 [11:34:36.876] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:30aadca22d41dc45 epochNumber=1 {"provingJobId":"1:BASE_PARITY:30aadca22d41dc45"} 11:34:36 [11:34:36.877] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testz70OML/1 with map size 1073741824 11:34:36 [11:34:36.878] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:36 [11:34:36.897] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a92f44809ed7f635 epochNumber=2 {"provingJobId":"1:BASE_PARITY:a92f44809ed7f635"} 11:34:36 [11:34:36.898] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testz70OML/2 with map size 1073741824 11:34:36 [11:34:36.901] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:37 [11:34:37.921] INFO: prover-client:proving-broker Proving Broker started 11:34:37 [11:34:37.926] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:30aadca22d41dc45 settled=false {"provingJobId":"1:BASE_PARITY:30aadca22d41dc45","status":"pending"} 11:34:37 [11:34:37.929] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a92f44809ed7f635 settled=false {"provingJobId":"1:BASE_PARITY:a92f44809ed7f635","status":"pending"} 11:34:37 [11:34:37.930] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:30aadca22d41dc45 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:30aadca22d41dc45"} 11:34:37 [11:34:37.931] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:30aadca22d41dc45 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:30aadca22d41dc45"} 11:34:37 [11:34:37.941] INFO: prover-client:proving-broker Proving Broker started 11:34:37 [11:34:37.942] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:86ac9af31492ba81 epochNumber=1 {"provingJobId":"1:BASE_PARITY:86ac9af31492ba81"} 11:34:37 [11:34:37.943] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaUhYcT/1 with map size 1073741824 11:34:37 [11:34:37.945] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:37 [11:34:37.970] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dfbd40dae0deb2fd epochNumber=2 {"provingJobId":"1:BASE_PARITY:dfbd40dae0deb2fd"} 11:34:37 [11:34:37.972] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testaUhYcT/2 with map size 1073741824 11:34:37 [11:34:37.975] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:37 [11:34:37.992] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:86ac9af31492ba81 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:86ac9af31492ba81"} 11:34:38 [11:34:38.000] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dfbd40dae0deb2fd type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:dfbd40dae0deb2fd"} 11:34:38 [11:34:38.014] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.015] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:81479d825dadb845 epochNumber=1 {"provingJobId":"1:BASE_PARITY:81479d825dadb845"} 11:34:38 [11:34:38.016] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test91CXOm/1 with map size 1073741824 11:34:38 [11:34:38.018] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.043] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5bb7eb388d0dd3e2 epochNumber=2 {"provingJobId":"1:BASE_PARITY:5bb7eb388d0dd3e2"} 11:34:38 [11:34:38.044] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test91CXOm/2 with map size 1073741824 11:34:38 [11:34:38.051] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.070] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:81479d825dadb845 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:81479d825dadb845"} 11:34:38 [11:34:38.071] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:81479d825dadb845 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:81479d825dadb845"} 11:34:38 [11:34:38.078] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5bb7eb388d0dd3e2 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:5bb7eb388d0dd3e2"} 11:34:38 [11:34:38.090] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.092] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c7e4f7aeabdfccde err=test error {"provingJoId":"1:BASE_PARITY:c7e4f7aeabdfccde"} 11:34:38 [11:34:38.098] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.099] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d2fe25ba2b8baa53 not found {"provingJobId":"1:BASE_PARITY:d2fe25ba2b8baa53"} 11:34:38 [11:34:38.104] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.105] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d772ac593f63a56f epochNumber=1 {"provingJobId":"1:BASE_PARITY:d772ac593f63a56f"} 11:34:38 [11:34:38.106] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZERqb4/1 with map size 1073741824 11:34:38 [11:34:38.109] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.139] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.141] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ebcf5db694417276 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ebcf5db694417276"} 11:34:38 [11:34:38.143] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testp6i6Gv/1 with map size 1073741824 11:34:38 [11:34:38.149] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.269] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ebcf5db694417276 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:ebcf5db694417276"} 11:34:38 [11:34:38.277] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.277] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:396352f49aa4597d epochNumber=1 {"provingJobId":"1:BASE_PARITY:396352f49aa4597d"} 11:34:38 [11:34:38.278] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvQJy4b/1 with map size 1073741824 11:34:38 [11:34:38.280] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.408] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:396352f49aa4597d timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:396352f49aa4597d"} 11:34:38 [11:34:38.410] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:12f9913a038c1ca0 epochNumber=10 {"provingJobId":"1:BASE_PARITY:12f9913a038c1ca0"} 11:34:38 [11:34:38.410] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testvQJy4b/10 with map size 1073741824 11:34:38 [11:34:38.412] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.459] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:38 [11:34:38.460] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 11:34:38 [11:34:38.565] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.566] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:14b545087386f382 epochNumber=1 {"provingJobId":"1:BASE_PARITY:14b545087386f382"} 11:34:38 [11:34:38.567] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testv4rt1k/1 with map size 1073741824 11:34:38 [11:34:38.569] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.744] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:14b545087386f382 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:14b545087386f382"} 11:34:38 [11:34:38.790] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.791] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6f0f6d6aa85fb2cc epochNumber=1 {"provingJobId":"1:BASE_PARITY:6f0f6d6aa85fb2cc"} 11:34:38 [11:34:38.793] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQvenvB/1 with map size 1073741824 11:34:38 [11:34:38.797] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.821] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6f0f6d6aa85fb2cc type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:6f0f6d6aa85fb2cc"} 11:34:38 [11:34:38.831] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.831] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f28fc85abbc7a62a epochNumber=1 {"provingJobId":"1:BASE_PARITY:f28fc85abbc7a62a"} 11:34:38 [11:34:38.832] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuPOeqh/1 with map size 1073741824 11:34:38 [11:34:38.835] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.854] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f28fc85abbc7a62a type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:f28fc85abbc7a62a"} 11:34:38 [11:34:38.855] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f28fc85abbc7a62a type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:f28fc85abbc7a62a"} 11:34:38 [11:34:38.865] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.866] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c223675063e1f52a epochNumber=1 {"provingJobId":"1:BASE_PARITY:c223675063e1f52a"} 11:34:38 [11:34:38.868] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5h1DKA/1 with map size 1073741824 11:34:38 [11:34:38.873] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.895] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c223675063e1f52a type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:c223675063e1f52a"} 11:34:38 [11:34:38.906] INFO: prover-client:proving-broker Proving Broker started 11:34:38 [11:34:38.907] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d2c68cc94e51b6b1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d2c68cc94e51b6b1"} 11:34:38 [11:34:38.907] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFJXm6m/1 with map size 1073741824 11:34:38 [11:34:38.912] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.939] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d2c68cc94e51b6b1 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:d2c68cc94e51b6b1"} 11:34:38 [11:34:38.940] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:177159415afeba21 epochNumber=3 {"provingJobId":"1:BASE_PARITY:177159415afeba21"} 11:34:38 [11:34:38.942] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testFJXm6m/3 with map size 1073741824 11:34:38 [11:34:38.947] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:38 [11:34:38.984] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:38 [11:34:38.984] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 11:34:38 [11:34:38.995] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d2c68cc94e51b6b1 err=test error {"provingJoId":"1:BASE_PARITY:d2c68cc94e51b6b1"} 11:34:38 [11:34:38.999] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdpvJtk/1 with map size 1073741824 11:34:39 [11:34:39.000] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.024] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testdpvJtk/2 with map size 1073741824 11:34:39 [11:34:39.030] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.061] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.063] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4c9dff4e81800083 settled=false {"provingJobId":"1:BASE_PARITY:4c9dff4e81800083","status":"pending"} 11:34:39 [11:34:39.065] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c8a740e542f2ceae settled=false {"provingJobId":"1:BASE_PARITY:c8a740e542f2ceae","status":"pending"} 11:34:39 [11:34:39.071] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUNYAe5/1 with map size 1073741824 11:34:39 [11:34:39.074] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.093] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUNYAe5/2 with map size 1073741824 11:34:39 [11:34:39.099] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.143] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.146] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9f8afc576c78a57c settled=true {"provingJobId":"1:BASE_PARITY:9f8afc576c78a57c","status":"fulfilled"} 11:34:39 [11:34:39.149] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:14594190df3da398 settled=true {"provingJobId":"2:BASE_PARITY:14594190df3da398","status":"fulfilled"} 11:34:39 [11:34:39.160] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcSgmIR/1 with map size 1073741824 11:34:39 [11:34:39.162] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.198] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcSgmIR/2 with map size 1073741824 11:34:39 [11:34:39.201] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.223] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.225] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2788e05831f844d7 settled=true {"provingJobId":"1:BASE_PARITY:2788e05831f844d7","status":"fulfilled"} 11:34:39 [11:34:39.227] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c76a9fc5d872b062 settled=false {"provingJobId":"1:BASE_PARITY:c76a9fc5d872b062","status":"pending"} 11:34:39 [11:34:39.241] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.242] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7f5ab32cb44d0abc epochNumber=1 {"provingJobId":"1:BASE_PARITY:7f5ab32cb44d0abc"} 11:34:39 [11:34:39.244] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6FUE6j/1 with map size 1073741824 11:34:39 [11:34:39.249] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.280] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.281] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f4025a60c90f5321 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f4025a60c90f5321"} 11:34:39 [11:34:39.282] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:f4025a60c90f5321: Error: db error: Error: db error 11:34:39 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:f4025a60c90f5321"} 11:34:39 [11:34:39.288] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.288] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f76cb6548820f6c3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f76cb6548820f6c3"} 11:34:39 [11:34:39.290] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUNQOHP/1 with map size 1073741824 11:34:39 [11:34:39.292] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.323] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f76cb6548820f6c3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f76cb6548820f6c3"} 11:34:39 [11:34:39.324] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f76cb6548820f6c3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f76cb6548820f6c3"} 11:34:39 [11:34:39.339] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.339] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f6bfdb52e6fcbf2b epochNumber=1 {"provingJobId":"1:BASE_PARITY:f6bfdb52e6fcbf2b"} 11:34:39 [11:34:39.341] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpzyHmm/1 with map size 1073741824 11:34:39 [11:34:39.345] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.370] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f6bfdb52e6fcbf2b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f6bfdb52e6fcbf2b"} 11:34:39 [11:34:39.370] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f6bfdb52e6fcbf2b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f6bfdb52e6fcbf2b"} 11:34:39 [11:34:39.371] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:f6bfdb52e6fcbf2b: Error: db error 11:34:39 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:f6bfdb52e6fcbf2b"} 11:34:39 [11:34:39.377] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.377] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b0ae3c761ec43170 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b0ae3c761ec43170"} 11:34:39 [11:34:39.379] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtUnDfS/1 with map size 1073741824 11:34:39 [11:34:39.384] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.409] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b0ae3c761ec43170 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b0ae3c761ec43170"} 11:34:39 [11:34:39.409] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b0ae3c761ec43170 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b0ae3c761ec43170"} 11:34:39 [11:34:39.421] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.422] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:24ed0db4569ba016 epochNumber=1 {"provingJobId":"1:BASE_PARITY:24ed0db4569ba016"} 11:34:39 [11:34:39.423] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKbMfi1/1 with map size 1073741824 11:34:39 [11:34:39.426] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.455] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:24ed0db4569ba016 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:24ed0db4569ba016"} 11:34:39 [11:34:39.456] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:24ed0db4569ba016 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:24ed0db4569ba016"} 11:34:39 [11:34:39.457] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:24ed0db4569ba016 jobErr=test error: Error: db error 11:34:39 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:24ed0db4569ba016"} 11:34:39 [11:34:39.464] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.466] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:30d4fd76d80acd37 not found {"provingJobId":"1:BASE_PARITY:30d4fd76d80acd37"} 11:34:39 [11:34:39.471] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.472] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:327b61d33aee8aeb err=test error {"provingJoId":"1:BASE_PARITY:327b61d33aee8aeb"} 11:34:39 [11:34:39.477] INFO: prover-client:proving-broker Proving Broker started 11:34:39 [11:34:39.488] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1fbe53a5962dc4b7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1fbe53a5962dc4b7"} 11:34:39 [11:34:39.491] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpzAiks/1 with map size 1073741824 11:34:39 [11:34:39.495] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.520] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1fbe53a5962dc4b7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:1fbe53a5962dc4b7"} 11:34:39 [11:34:39.521] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1fbe53a5962dc4b7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:1fbe53a5962dc4b7"} 11:34:39 [11:34:39.543] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:066b85b6939ae49f epochNumber=2 {"provingJobId":"2:BASE_PARITY:066b85b6939ae49f"} 11:34:39 [11:34:39.547] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpzAiks/2 with map size 1073741824 11:34:39 [11:34:39.556] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.574] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:066b85b6939ae49f type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:066b85b6939ae49f"} 11:34:39 [11:34:39.574] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:066b85b6939ae49f type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:066b85b6939ae49f"} 11:34:39 [11:34:39.643] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:cf67b367fbad31bf epochNumber=3 {"provingJobId":"3:BASE_PARITY:cf67b367fbad31bf"} 11:34:39 [11:34:39.644] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testpzAiks/3 with map size 1073741824 11:34:39 [11:34:39.652] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.683] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:39 [11:34:39.684] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 11:34:39 [11:34:39.742] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:4aca560a67ec205d epochNumber=4 {"provingJobId":"4:BASE_PARITY:4aca560a67ec205d"} 11:34:39 [11:34:39.743] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testpzAiks/4 with map size 1073741824 11:34:39 [11:34:39.748] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.788] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:39 [11:34:39.789] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 11:34:39 [11:34:39.833] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:36a922367983d9aa epochNumber=5 {"provingJobId":"5:BASE_PARITY:36a922367983d9aa"} 11:34:39 [11:34:39.834] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testpzAiks/5 with map size 1073741824 11:34:39 [11:34:39.838] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 11:34:39 [11:34:39.869] INFO: prover-client:proving-broker Cleaned up jobs=1 11:34:39 [11:34:39.870] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 11:34:39 PASS src/proving_broker/proving_broker.test.ts (13.666 s) 11:34:39 ProvingBroker 11:34:39 Producer API 11:34:39 ✓ refuses stale jobs (42 ms) 11:34:39 ✓ enqueues jobs (5 ms) 11:34:39 ✓ ignores duplicate jobs (5 ms) 11:34:39 ✓ reports correct status when enqueuing repeat jobs (8 ms) 11:34:39 ✓ throws an error in case of duplicate job IDs (30 ms) 11:34:39 ✓ returns not-found status for non-existing jobs (2 ms) 11:34:39 ✓ cancels jobs in queue (4 ms) 11:34:39 ✓ cancels jobs in-progress (2 ms) 11:34:39 ✓ returns job result if successful (2 ms) 11:34:39 ✓ returns job error if failed (2 ms) 11:34:39 ✓ correctly returns job status for concurrent writes (2 ms) 11:34:39 ✓ refuses stale jobs (47 ms) 11:34:39 ✓ enqueues jobs (36 ms) 11:34:39 ✓ ignores duplicate jobs (33 ms) 11:34:39 ✓ reports correct status when enqueuing repeat jobs (39 ms) 11:34:39 ✓ throws an error in case of duplicate job IDs (26 ms) 11:34:39 ✓ returns not-found status for non-existing jobs (4 ms) 11:34:39 ✓ cancels jobs in queue (34 ms) 11:34:39 ✓ cancels jobs in-progress (50 ms) 11:34:39 ✓ returns job result if successful (41 ms) 11:34:39 ✓ returns job error if failed (36 ms) 11:34:39 ✓ correctly returns job status for concurrent writes (35 ms) 11:34:39 Consumer API 11:34:39 ✓ returns undefined if no jobs are available (1 ms) 11:34:39 ✓ returns jobs in priority order (1 ms) 11:34:39 ✓ returns undefined if no jobs are available for the given allowList (11 ms) 11:34:39 ✓ returns a job if it is in the allowList (2 ms) 11:34:39 ✓ returns the most important job if it is in the allowList (3 ms) 11:34:39 ✓ returns any job if filter is empty (2 ms) 11:34:39 ✓ returns a new job when reporting job success (2 ms) 11:34:39 ✓ returns a new job when reporting permanent error (2 ms) 11:34:39 ✓ returns a new job when reporting retry-able error (2 ms) 11:34:39 ✓ returns a new job when reporting progress if current one is cancelled (2 ms) 11:34:39 ✓ returns a new job if job is already in progress elsewhere (1057 ms) 11:34:39 ✓ avoids sending the same job to a new agent after a restart (1007 ms) 11:34:39 ✓ avoids sending a completed job to a new agent after a restart (1007 ms) 11:34:39 ✓ tracks job result if in progress (4 ms) 11:34:39 ✓ tracks job result even if job is in queue (3 ms) 11:34:39 ✓ ignores reported job error if unknown job (2 ms) 11:34:39 ✓ ignores job result if unknown job (2 ms) 11:34:39 ✓ returns undefined if no jobs are available (2 ms) 11:34:39 ✓ returns jobs in priority order (51 ms) 11:34:39 ✓ returns undefined if no jobs are available for the given allowList (28 ms) 11:34:39 ✓ returns a job if it is in the allowList (63 ms) 11:34:39 ✓ returns the most important job if it is in the allowList (66 ms) 11:34:39 ✓ returns any job if filter is empty (66 ms) 11:34:39 ✓ returns a new job when reporting job success (41 ms) 11:34:39 ✓ returns a new job when reporting permanent error (42 ms) 11:34:39 ✓ returns a new job when reporting retry-able error (35 ms) 11:34:39 ✓ returns a new job when reporting progress if current one is cancelled (33 ms) 11:34:39 ✓ returns a new job if job is already in progress elsewhere (1106 ms) 11:34:39 ✓ avoids sending the same job to a new agent after a restart (1051 ms) 11:34:39 ✓ avoids sending a completed job to a new agent after a restart (1064 ms) 11:34:39 ✓ tracks job result if in progress (72 ms) 11:34:39 ✓ tracks job result even if job is in queue (76 ms) 11:34:39 ✓ ignores reported job error if unknown job (5 ms) 11:34:39 ✓ ignores job result if unknown job (8 ms) 11:34:39 Timeouts 11:34:39 ✓ tracks in progress jobs (2 ms) 11:34:39 ✓ re-enqueues jobs that time out (109 ms) 11:34:39 ✓ cancel stale jobs that time out (233 ms) 11:34:39 ✓ keeps the jobs in progress while it is alive (204 ms) 11:34:39 ✓ tracks in progress jobs (35 ms) 11:34:39 ✓ re-enqueues jobs that time out (139 ms) 11:34:39 ✓ cancel stale jobs that time out (288 ms) 11:34:39 ✓ keeps the jobs in progress while it is alive (224 ms) 11:34:39 Retries 11:34:39 ✓ retries jobs (3 ms) 11:34:39 ✓ retries up to a maximum number of times (2 ms) 11:34:39 ✓ passing retry=false does not retry (3 ms) 11:34:39 ✓ does not retry if job is stale (29 ms) 11:34:39 ✓ retries jobs (36 ms) 11:34:39 ✓ retries up to a maximum number of times (36 ms) 11:34:39 ✓ passing retry=false does not retry (40 ms) 11:34:39 ✓ does not retry if job is stale (92 ms) 11:34:39 Database management 11:34:39 ✓ re-enqueues proof requests on start (3 ms) 11:34:39 ✓ restores proof results on start (3 ms) 11:34:39 ✓ only re-enqueues unfinished jobs (2 ms) 11:34:39 ✓ saves job when enqueued (3 ms) 11:34:39 ✓ does not retain job if database fails to save (2 ms) 11:34:39 ✓ saves job result (2 ms) 11:34:39 ✓ saves result even if database fails to save (3 ms) 11:34:39 ✓ saves job error (2 ms) 11:34:39 ✓ saves job error even if database fails to save (8 ms) 11:34:39 ✓ does not save job result if job is unknown (2 ms) 11:34:39 ✓ does not save job error if job is unknown (4 ms) 11:34:39 ✓ cleans up old jobs periodically (272 ms) 11:34:39 ✓ re-enqueues proof requests on start (71 ms) 11:34:39 ✓ restores proof results on start (85 ms) 11:34:39 ✓ only re-enqueues unfinished jobs (77 ms) 11:34:39 ✓ saves job when enqueued (45 ms) 11:34:39 ✓ does not retain job if database fails to save (5 ms) 11:34:39 ✓ saves job result (52 ms) 11:34:39 ✓ saves result even if database fails to save (37 ms) 11:34:39 ✓ saves job error (45 ms) 11:34:39 ✓ saves job error even if database fails to save (42 ms) 11:34:39 ✓ does not save job result if job is unknown (5 ms) 11:34:39 ✓ does not save job error if job is unknown (6 ms) 11:34:39 ✓ cleans up old jobs periodically (442 ms) 11:34:39 11:34:39 Test Suites: 1 passed, 1 total 11:34:39 Tests: 96 passed, 96 total 11:34:39 Snapshots: 0 total 11:34:39 Time: 13.773 s 11:34:39 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 11:34:39 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?