ACI ·

Parent Log: http://ci.aztec-labs.com/025889bb38746a8e Command: 02573248fd9c66ba yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/712835e3cfa2ed6f980f2d482aa4b0ae767046a7 Env: REF_NAME=gh-readonly-queue/next/pr-15077-4b0e4dc09fa55f0bc808fa0c7c3e95e8fa997069 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Tue Jun 17 21:08:00 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15077_amd64_x3-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 21:08:04 [21:08:04.823] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.829] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4c991c0b00c2c6d3 epochNumber=42 {"provingJobId":"1:BASE_PARITY:4c991c0b00c2c6d3"} 21:08:04 [21:08:04.832] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:226ad112aa6cbf62 epochNumber=1 {"provingJobId":"1:BASE_PARITY:226ad112aa6cbf62"} 21:08:04 [21:08:04.860] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.861] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2445d587c550d95e epochNumber=1 {"provingJobId":"1:BASE_PARITY:2445d587c550d95e"} 21:08:04 [21:08:04.863] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:31506ce4c5963264 epochNumber=1 {"provingJobId":"1:BASE_PARITY:31506ce4c5963264"} 21:08:04 [21:08:04.865] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.866] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:35d9ac829543eb0d epochNumber=1 {"provingJobId":"1:BASE_PARITY:35d9ac829543eb0d"} 21:08:04 [21:08:04.867] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:35d9ac829543eb0d epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:35d9ac829543eb0d"} 21:08:04 [21:08:04.869] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.870] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0934443bf7e5931c epochNumber=1 {"provingJobId":"1:BASE_PARITY:0934443bf7e5931c"} 21:08:04 [21:08:04.872] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0934443bf7e5931c epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:0934443bf7e5931c"} 21:08:04 [21:08:04.872] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0934443bf7e5931c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0934443bf7e5931c"} 21:08:04 [21:08:04.873] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0934443bf7e5931c epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:0934443bf7e5931c"} 21:08:04 [21:08:04.874] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.875] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:59629ccfdeae2bd0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:59629ccfdeae2bd0"} 21:08:04 [21:08:04.901] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.903] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.903] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8c35236e1fba8abb epochNumber=1 {"provingJobId":"1:BASE_PARITY:8c35236e1fba8abb"} 21:08:04 [21:08:04.904] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:8c35236e1fba8abb {"provingJobId":"1:BASE_PARITY:8c35236e1fba8abb"} 21:08:04 [21:08:04.904] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8c35236e1fba8abb type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8c35236e1fba8abb"} 21:08:04 [21:08:04.904] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8c35236e1fba8abb type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:8c35236e1fba8abb"} 21:08:04 [21:08:04.906] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.906] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:52c79678409f654b epochNumber=1 {"provingJobId":"1:BASE_PARITY:52c79678409f654b"} 21:08:04 [21:08:04.906] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:52c79678409f654b {"provingJobId":"1:BASE_PARITY:52c79678409f654b"} 21:08:04 [21:08:04.907] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:52c79678409f654b type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:52c79678409f654b"} 21:08:04 [21:08:04.908] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.909] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:26d2d2a068b958e9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:26d2d2a068b958e9"} 21:08:04 [21:08:04.909] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:26d2d2a068b958e9 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:26d2d2a068b958e9"} 21:08:04 [21:08:04.909] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:26d2d2a068b958e9 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:26d2d2a068b958e9"} 21:08:04 [21:08:04.910] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.911] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:35ec398d13e1059a epochNumber=1 {"provingJobId":"1:BASE_PARITY:35ec398d13e1059a"} 21:08:04 [21:08:04.911] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:35ec398d13e1059a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:35ec398d13e1059a"} 21:08:04 [21:08:04.911] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:35ec398d13e1059a type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:35ec398d13e1059a"} 21:08:04 [21:08:04.912] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.912] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b5c6811e9ea49b1b epochNumber=0 {"provingJobId":"1:BASE_PARITY:b5c6811e9ea49b1b"} 21:08:04 [21:08:04.912] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b5c6811e9ea49b1b epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b5c6811e9ea49b1b"} 21:08:04 [21:08:04.913] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5fd65f8b4ab23efc epochNumber=0 {"provingJobId":"1:BASE_PARITY:5fd65f8b4ab23efc"} 21:08:04 [21:08:04.913] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b5c6811e9ea49b1b epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b5c6811e9ea49b1b"} 21:08:04 [21:08:04.913] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dd1ad42403f0e4be epochNumber=0 {"provingJobId":"1:BASE_PARITY:dd1ad42403f0e4be"} 21:08:04 [21:08:04.921] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.922] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.923] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:19e8888c751e9b43 epochNumber=2 {"provingJobId":"1:BASE_PARITY:19e8888c751e9b43"} 21:08:04 [21:08:04.923] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f1e21f9ca91f985d epochNumber=1 {"provingJobId":"1:BASE_PARITY:f1e21f9ca91f985d"} 21:08:04 [21:08:04.924] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.925] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bc0ebc28708a8411 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bc0ebc28708a8411"} 21:08:04 [21:08:04.926] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.926] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b8a658e3d6bc7650 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b8a658e3d6bc7650"} 21:08:04 [21:08:04.926] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a7b4b7c142a836b6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a7b4b7c142a836b6"} 21:08:04 [21:08:04.926] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:74deb03986626ff1 epochNumber=2 {"provingJobId":"1:BASE_PARITY:74deb03986626ff1"} 21:08:04 [21:08:04.927] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:35a042acb40e9469 epochNumber=1 {"provingJobId":"1:BASE_PARITY:35a042acb40e9469"} 21:08:04 [21:08:04.927] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.928] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fe88b31ecc78e72a epochNumber=1 {"provingJobId":"1:BASE_PARITY:fe88b31ecc78e72a"} 21:08:04 [21:08:04.928] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:143e1ce7a4f1f472 epochNumber=1 {"provingJobId":"1:BASE_PARITY:143e1ce7a4f1f472"} 21:08:04 [21:08:04.928] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:26a45532aed0d087 epochNumber=2 {"provingJobId":"1:BASE_PARITY:26a45532aed0d087"} 21:08:04 [21:08:04.928] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2249dff84467cfdc epochNumber=1 {"provingJobId":"1:BASE_PARITY:2249dff84467cfdc"} 21:08:04 [21:08:04.929] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.930] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:19aef1c35a7337fd epochNumber=1 {"provingJobId":"1:BASE_PARITY:19aef1c35a7337fd"} 21:08:04 [21:08:04.930] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7659189a3caa35e2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7659189a3caa35e2"} 21:08:04 [21:08:04.930] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a9b67a5948d149f0 epochNumber=2 {"provingJobId":"1:BASE_PARITY:a9b67a5948d149f0"} 21:08:04 [21:08:04.931] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cb548e8a1a3eb226 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cb548e8a1a3eb226"} 21:08:04 [21:08:04.932] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.932] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eca1f03ec3c6375c epochNumber=1 {"provingJobId":"1:BASE_PARITY:eca1f03ec3c6375c"} 21:08:04 [21:08:04.932] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8f021af0b5dca516 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8f021af0b5dca516"} 21:08:04 [21:08:04.933] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:eca1f03ec3c6375c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:eca1f03ec3c6375c"} 21:08:04 [21:08:04.934] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.934] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b2fd496b4bd8829e epochNumber=1 {"provingJobId":"1:BASE_PARITY:b2fd496b4bd8829e"} 21:08:04 [21:08:04.934] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f67f31ce9614b4be epochNumber=1 {"provingJobId":"1:BASE_PARITY:f67f31ce9614b4be"} 21:08:04 [21:08:04.935] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b2fd496b4bd8829e type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:b2fd496b4bd8829e"} 21:08:04 [21:08:04.935] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.936] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d77c0c47a3dfd225 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d77c0c47a3dfd225"} 21:08:04 [21:08:04.936] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7c19cf6ccefdb1bb epochNumber=1 {"provingJobId":"1:BASE_PARITY:7c19cf6ccefdb1bb"} 21:08:04 [21:08:04.936] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d77c0c47a3dfd225 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:d77c0c47a3dfd225"} 21:08:04 [21:08:04.937] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.938] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7db77d2a20a9bf1f epochNumber=1 {"provingJobId":"1:BASE_PARITY:7db77d2a20a9bf1f"} 21:08:04 [21:08:04.938] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7db77d2a20a9bf1f {"provingJobId":"1:BASE_PARITY:7db77d2a20a9bf1f"} 21:08:04 [21:08:04.938] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7db77d2a20a9bf1f type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:7db77d2a20a9bf1f"} 21:08:04 [21:08:04.939] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dc0cba62d7530411 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dc0cba62d7530411"} 21:08:04 [21:08:04.939] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7db77d2a20a9bf1f has already been completed {"provingJobId":"1:BASE_PARITY:7db77d2a20a9bf1f"} 21:08:04 [21:08:04.940] INFO: prover-client:proving-broker Proving Broker started 21:08:04 [21:08:04.941] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6e3cf471c8fb61c9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6e3cf471c8fb61c9"} 21:08:04 [21:08:04.941] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5ebc3a30e7b6075e epochNumber=2 {"provingJobId":"1:BASE_PARITY:5ebc3a30e7b6075e"} 21:08:05 [21:08:05.993] INFO: prover-client:proving-broker Proving Broker started 21:08:05 [21:08:05.994] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6e3cf471c8fb61c9 settled=false {"provingJobId":"1:BASE_PARITY:6e3cf471c8fb61c9","status":"pending"} 21:08:05 [21:08:05.994] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5ebc3a30e7b6075e settled=false {"provingJobId":"1:BASE_PARITY:5ebc3a30e7b6075e","status":"pending"} 21:08:05 [21:08:05.995] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:6e3cf471c8fb61c9 type=BASE_PARITY startedAt=1750194484941 older agent has taken job {"provingJobId":"1:BASE_PARITY:6e3cf471c8fb61c9"} 21:08:05 [21:08:05.995] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6e3cf471c8fb61c9 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:6e3cf471c8fb61c9"} 21:08:05 [21:08:05.996] INFO: prover-client:proving-broker Proving Broker started 21:08:05 [21:08:05.997] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b84e3c88c6a19467 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b84e3c88c6a19467"} 21:08:05 [21:08:05.997] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c399659629496a11 epochNumber=2 {"provingJobId":"1:BASE_PARITY:c399659629496a11"} 21:08:06 [21:08:06.998] INFO: prover-client:proving-broker Proving Broker started 21:08:06 [21:08:06.999] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b84e3c88c6a19467 settled=false {"provingJobId":"1:BASE_PARITY:b84e3c88c6a19467","status":"pending"} 21:08:06 [21:08:06.999] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c399659629496a11 settled=false {"provingJobId":"1:BASE_PARITY:c399659629496a11","status":"pending"} 21:08:06 [21:08:06.999] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b84e3c88c6a19467 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:b84e3c88c6a19467"} 21:08:07 [21:08:07.001] INFO: prover-client:proving-broker Proving Broker started 21:08:07 [21:08:07.001] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2317f319da86ab4f epochNumber=1 {"provingJobId":"1:BASE_PARITY:2317f319da86ab4f"} 21:08:07 [21:08:07.002] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2bbd44bc14b01060 epochNumber=2 {"provingJobId":"1:BASE_PARITY:2bbd44bc14b01060"} 21:08:08 [21:08:08.002] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.003] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2317f319da86ab4f settled=false {"provingJobId":"1:BASE_PARITY:2317f319da86ab4f","status":"pending"} 21:08:08 [21:08:08.003] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2bbd44bc14b01060 settled=false {"provingJobId":"1:BASE_PARITY:2bbd44bc14b01060","status":"pending"} 21:08:08 [21:08:08.004] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2317f319da86ab4f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2317f319da86ab4f"} 21:08:08 [21:08:08.004] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2317f319da86ab4f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2317f319da86ab4f"} 21:08:08 [21:08:08.006] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.006] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7a4bb79e5c07e2af epochNumber=1 {"provingJobId":"1:BASE_PARITY:7a4bb79e5c07e2af"} 21:08:08 [21:08:08.006] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:22206fabe92bbd00 epochNumber=2 {"provingJobId":"1:BASE_PARITY:22206fabe92bbd00"} 21:08:08 [21:08:08.007] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7a4bb79e5c07e2af type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:7a4bb79e5c07e2af"} 21:08:08 [21:08:08.007] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:22206fabe92bbd00 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:22206fabe92bbd00"} 21:08:08 [21:08:08.008] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.009] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ff88c5b077739eb8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ff88c5b077739eb8"} 21:08:08 [21:08:08.010] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0a16e7dede82f3f9 epochNumber=2 {"provingJobId":"1:BASE_PARITY:0a16e7dede82f3f9"} 21:08:08 [21:08:08.011] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ff88c5b077739eb8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ff88c5b077739eb8"} 21:08:08 [21:08:08.011] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ff88c5b077739eb8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ff88c5b077739eb8"} 21:08:08 [21:08:08.012] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0a16e7dede82f3f9 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:0a16e7dede82f3f9"} 21:08:08 [21:08:08.014] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.015] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:77ad7e71abae7b0b err=test error {"provingJoId":"1:BASE_PARITY:77ad7e71abae7b0b"} 21:08:08 [21:08:08.017] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.018] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:54647d83193957c8 not found {"provingJobId":"1:BASE_PARITY:54647d83193957c8"} 21:08:08 [21:08:08.020] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.020] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2865b8a6d4aef918 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2865b8a6d4aef918"} 21:08:08 [21:08:08.022] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.023] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03597f9689626555 epochNumber=1 {"provingJobId":"1:BASE_PARITY:03597f9689626555"} 21:08:08 [21:08:08.125] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:03597f9689626555 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:03597f9689626555"} 21:08:08 [21:08:08.132] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.133] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e753c36802d13740 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e753c36802d13740"} 21:08:08 [21:08:08.235] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e753c36802d13740 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:e753c36802d13740"} 21:08:08 [21:08:08.241] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f5dc3b41d32d4d35 epochNumber=10 {"provingJobId":"1:BASE_PARITY:f5dc3b41d32d4d35"} 21:08:08 [21:08:08.262] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:08 [21:08:08.369] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.371] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:680a79ee727d62d3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:680a79ee727d62d3"} 21:08:08 [21:08:08.524] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:680a79ee727d62d3 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:680a79ee727d62d3"} 21:08:08 [21:08:08.578] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.581] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ec5ec86e768b3923 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ec5ec86e768b3923"} 21:08:08 [21:08:08.585] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ec5ec86e768b3923 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:ec5ec86e768b3923"} 21:08:08 [21:08:08.588] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.590] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1410c193cfa8255c epochNumber=1 {"provingJobId":"1:BASE_PARITY:1410c193cfa8255c"} 21:08:08 [21:08:08.594] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1410c193cfa8255c type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:1410c193cfa8255c"} 21:08:08 [21:08:08.595] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1410c193cfa8255c type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:1410c193cfa8255c"} 21:08:08 [21:08:08.596] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.597] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a16930e714a9bfe1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a16930e714a9bfe1"} 21:08:08 [21:08:08.597] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a16930e714a9bfe1 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:a16930e714a9bfe1"} 21:08:08 [21:08:08.598] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.599] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:73984c8ef671a2fc epochNumber=1 {"provingJobId":"1:BASE_PARITY:73984c8ef671a2fc"} 21:08:08 [21:08:08.600] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:73984c8ef671a2fc type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:73984c8ef671a2fc"} 21:08:08 [21:08:08.600] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:15c8723e4c68431d epochNumber=3 {"provingJobId":"1:BASE_PARITY:15c8723e4c68431d"} 21:08:08 [21:08:08.624] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:08 [21:08:08.626] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:73984c8ef671a2fc err=test error {"provingJoId":"1:BASE_PARITY:73984c8ef671a2fc"} 21:08:08 [21:08:08.629] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.629] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:77a34e9f63e443a9 settled=false {"provingJobId":"1:BASE_PARITY:77a34e9f63e443a9","status":"pending"} 21:08:08 [21:08:08.629] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3a5cea7a7e15b639 settled=false {"provingJobId":"1:BASE_PARITY:3a5cea7a7e15b639","status":"pending"} 21:08:08 [21:08:08.634] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.635] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5da7238bad3f54e0 settled=true {"provingJobId":"1:BASE_PARITY:5da7238bad3f54e0","status":"fulfilled"} 21:08:08 [21:08:08.635] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:ae848ec3f1fd8430 settled=true {"provingJobId":"2:BASE_PARITY:ae848ec3f1fd8430","status":"fulfilled"} 21:08:08 [21:08:08.637] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.637] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:51fe7d32e3ba0b5d settled=true {"provingJobId":"1:BASE_PARITY:51fe7d32e3ba0b5d","status":"fulfilled"} 21:08:08 [21:08:08.637] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6684f4e6a6133bd2 settled=false {"provingJobId":"1:BASE_PARITY:6684f4e6a6133bd2","status":"pending"} 21:08:08 [21:08:08.638] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.640] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bf5826c86bee2015 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bf5826c86bee2015"} 21:08:08 [21:08:08.642] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.642] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d8699f392e40a39c epochNumber=1 {"provingJobId":"1:BASE_PARITY:d8699f392e40a39c"} 21:08:08 [21:08:08.643] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:d8699f392e40a39c: Error: db error: Error: db error 21:08:08 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:d8699f392e40a39c"} 21:08:08 [21:08:08.644] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.645] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:da9792c127834999 epochNumber=1 {"provingJobId":"1:BASE_PARITY:da9792c127834999"} 21:08:08 [21:08:08.645] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:da9792c127834999 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:da9792c127834999"} 21:08:08 [21:08:08.645] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:da9792c127834999 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:da9792c127834999"} 21:08:08 [21:08:08.647] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.647] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:14b9744c974f2a97 epochNumber=1 {"provingJobId":"1:BASE_PARITY:14b9744c974f2a97"} 21:08:08 [21:08:08.648] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:14b9744c974f2a97 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:14b9744c974f2a97"} 21:08:08 [21:08:08.648] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:14b9744c974f2a97 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:14b9744c974f2a97"} 21:08:08 [21:08:08.649] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:14b9744c974f2a97: Error: db error 21:08:08 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:14b9744c974f2a97"} 21:08:08 [21:08:08.651] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.652] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2493d23910d8ae7c epochNumber=1 {"provingJobId":"1:BASE_PARITY:2493d23910d8ae7c"} 21:08:08 [21:08:08.653] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2493d23910d8ae7c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2493d23910d8ae7c"} 21:08:08 [21:08:08.653] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2493d23910d8ae7c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:2493d23910d8ae7c"} 21:08:08 [21:08:08.657] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.659] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9c330095d797a3bb epochNumber=1 {"provingJobId":"1:BASE_PARITY:9c330095d797a3bb"} 21:08:08 [21:08:08.660] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9c330095d797a3bb type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9c330095d797a3bb"} 21:08:08 [21:08:08.660] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9c330095d797a3bb type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9c330095d797a3bb"} 21:08:08 [21:08:08.662] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:9c330095d797a3bb jobErr=test error: Error: db error 21:08:08 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:9c330095d797a3bb"} 21:08:08 [21:08:08.665] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.667] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a34c655bc8baf6c0 not found {"provingJobId":"1:BASE_PARITY:a34c655bc8baf6c0"} 21:08:08 [21:08:08.669] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.670] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:65b6da5573e75416 err=test error {"provingJoId":"1:BASE_PARITY:65b6da5573e75416"} 21:08:08 [21:08:08.672] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.684] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ede590c176ca4367 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ede590c176ca4367"} 21:08:08 [21:08:08.685] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ede590c176ca4367 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ede590c176ca4367"} 21:08:08 [21:08:08.685] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ede590c176ca4367 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ede590c176ca4367"} 21:08:08 [21:08:08.696] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:cb15369796f0b9e8 epochNumber=2 {"provingJobId":"2:BASE_PARITY:cb15369796f0b9e8"} 21:08:08 [21:08:08.697] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:cb15369796f0b9e8 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:cb15369796f0b9e8"} 21:08:08 [21:08:08.698] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:cb15369796f0b9e8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:cb15369796f0b9e8"} 21:08:08 [21:08:08.760] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:60cf64d9af2f44c5 epochNumber=3 {"provingJobId":"3:BASE_PARITY:60cf64d9af2f44c5"} 21:08:08 [21:08:08.777] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:08 [21:08:08.824] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:bc3f897c36c8d861 epochNumber=4 {"provingJobId":"4:BASE_PARITY:bc3f897c36c8d861"} 21:08:08 [21:08:08.830] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:08 [21:08:08.888] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:d817a60fd1434199 epochNumber=5 {"provingJobId":"5:BASE_PARITY:d817a60fd1434199"} 21:08:08 [21:08:08.914] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:08 [21:08:08.946] INFO: prover-client:proving-broker Proving Broker started 21:08:08 [21:08:08.948] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7ce948d5f69ddc24 epochNumber=42 {"provingJobId":"1:BASE_PARITY:7ce948d5f69ddc24"} 21:08:08 [21:08:08.949] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testW8FkPH/42 with map size 1073741824 21:08:08 [21:08:08.959] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:08.999] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:52110d4602bdb5eb epochNumber=1 {"provingJobId":"1:BASE_PARITY:52110d4602bdb5eb"} 21:08:09 [21:08:09.006] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.006] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2f45bee284dcb9d3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2f45bee284dcb9d3"} 21:08:09 [21:08:09.008] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWNZ6Bn/1 with map size 1073741824 21:08:09 [21:08:09.015] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.043] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c6f41bc21c908542 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c6f41bc21c908542"} 21:08:09 [21:08:09.061] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.062] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:06543570f438a82f epochNumber=1 {"provingJobId":"1:BASE_PARITY:06543570f438a82f"} 21:08:09 [21:08:09.066] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhCLD9v/1 with map size 1073741824 21:08:09 [21:08:09.072] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.095] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:06543570f438a82f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:06543570f438a82f"} 21:08:09 [21:08:09.100] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.100] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea99ccd9143b258f epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea99ccd9143b258f"} 21:08:09 [21:08:09.101] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5xS1vt/1 with map size 1073741824 21:08:09 [21:08:09.104] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.126] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ea99ccd9143b258f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:ea99ccd9143b258f"} 21:08:09 [21:08:09.128] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ea99ccd9143b258f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ea99ccd9143b258f"} 21:08:09 [21:08:09.136] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ea99ccd9143b258f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:ea99ccd9143b258f"} 21:08:09 [21:08:09.143] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.144] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:74bc288db017fac9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:74bc288db017fac9"} 21:08:09 [21:08:09.146] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8BaWcS/1 with map size 1073741824 21:08:09 [21:08:09.148] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.186] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.190] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.191] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:237822c952491b90 epochNumber=1 {"provingJobId":"1:BASE_PARITY:237822c952491b90"} 21:08:09 [21:08:09.193] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3DxCrL/1 with map size 1073741824 21:08:09 [21:08:09.199] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.218] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:237822c952491b90 {"provingJobId":"1:BASE_PARITY:237822c952491b90"} 21:08:09 [21:08:09.220] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:237822c952491b90 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:237822c952491b90"} 21:08:09 [21:08:09.221] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:237822c952491b90 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:237822c952491b90"} 21:08:09 [21:08:09.234] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.236] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:70c19da2ad4fb6bc epochNumber=1 {"provingJobId":"1:BASE_PARITY:70c19da2ad4fb6bc"} 21:08:09 [21:08:09.237] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0eBdOp/1 with map size 1073741824 21:08:09 [21:08:09.242] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.272] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:70c19da2ad4fb6bc {"provingJobId":"1:BASE_PARITY:70c19da2ad4fb6bc"} 21:08:09 [21:08:09.273] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:70c19da2ad4fb6bc type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:70c19da2ad4fb6bc"} 21:08:09 [21:08:09.288] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.290] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8a9e4ea738866fb3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8a9e4ea738866fb3"} 21:08:09 [21:08:09.294] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkiq4ys/1 with map size 1073741824 21:08:09 [21:08:09.296] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.316] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8a9e4ea738866fb3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8a9e4ea738866fb3"} 21:08:09 [21:08:09.317] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8a9e4ea738866fb3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8a9e4ea738866fb3"} 21:08:09 [21:08:09.328] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.329] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7e8d97667081b67d epochNumber=1 {"provingJobId":"1:BASE_PARITY:7e8d97667081b67d"} 21:08:09 [21:08:09.331] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testI3HV8m/1 with map size 1073741824 21:08:09 [21:08:09.335] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.358] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7e8d97667081b67d type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:7e8d97667081b67d"} 21:08:09 [21:08:09.359] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7e8d97667081b67d type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:7e8d97667081b67d"} 21:08:09 [21:08:09.372] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.373] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:15dc8c25fcded7f8 epochNumber=0 {"provingJobId":"1:BASE_PARITY:15dc8c25fcded7f8"} 21:08:09 [21:08:09.373] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testNuq7pq/0 with map size 1073741824 21:08:09 [21:08:09.375] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.398] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:15dc8c25fcded7f8 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:15dc8c25fcded7f8"} 21:08:09 [21:08:09.399] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:83ceb31cbfff4ace epochNumber=0 {"provingJobId":"1:BASE_PARITY:83ceb31cbfff4ace"} 21:08:09 [21:08:09.399] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:15dc8c25fcded7f8 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:15dc8c25fcded7f8"} 21:08:09 [21:08:09.400] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2e2f266c974ba909 epochNumber=0 {"provingJobId":"1:BASE_PARITY:2e2f266c974ba909"} 21:08:09 [21:08:09.416] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.421] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.423] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:020df5c8a8ca9ea7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:020df5c8a8ca9ea7"} 21:08:09 [21:08:09.424] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfJuI4u/2 with map size 1073741824 21:08:09 [21:08:09.426] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.447] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3f198f1fd98940a2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3f198f1fd98940a2"} 21:08:09 [21:08:09.448] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfJuI4u/1 with map size 1073741824 21:08:09 [21:08:09.451] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.483] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.484] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:32c20e18374ce9f0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:32c20e18374ce9f0"} 21:08:09 [21:08:09.485] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXr6Keo/1 with map size 1073741824 21:08:09 [21:08:09.488] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.510] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.510] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9a89c7e323a898b8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9a89c7e323a898b8"} 21:08:09 [21:08:09.511] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrVTpFM/1 with map size 1073741824 21:08:09 [21:08:09.514] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.533] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7e86a17eafbc3829 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7e86a17eafbc3829"} 21:08:09 [21:08:09.541] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:509c6b5078befc82 epochNumber=2 {"provingJobId":"1:BASE_PARITY:509c6b5078befc82"} 21:08:09 [21:08:09.542] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrVTpFM/2 with map size 1073741824 21:08:09 [21:08:09.544] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.578] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:17aa75412cc1377e epochNumber=1 {"provingJobId":"1:BASE_PARITY:17aa75412cc1377e"} 21:08:09 [21:08:09.591] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.592] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c2ab9b1e5b857077 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c2ab9b1e5b857077"} 21:08:09 [21:08:09.594] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrg4KKi/1 with map size 1073741824 21:08:09 [21:08:09.597] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.615] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cdc7e35a6cef42d3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cdc7e35a6cef42d3"} 21:08:09 [21:08:09.622] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c3eb268e69e704c5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:c3eb268e69e704c5"} 21:08:09 [21:08:09.623] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrg4KKi/2 with map size 1073741824 21:08:09 [21:08:09.625] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.647] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1068b5593b8f71e1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1068b5593b8f71e1"} 21:08:09 [21:08:09.660] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.665] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a8befb8f07c13b22 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a8befb8f07c13b22"} 21:08:09 [21:08:09.667] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvzexAp/1 with map size 1073741824 21:08:09 [21:08:09.676] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.697] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3d065148e2430441 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3d065148e2430441"} 21:08:09 [21:08:09.704] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7295b1cdddd8b0fe epochNumber=2 {"provingJobId":"1:BASE_PARITY:7295b1cdddd8b0fe"} 21:08:09 [21:08:09.705] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvzexAp/2 with map size 1073741824 21:08:09 [21:08:09.708] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.728] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cc63f0ab8d711bb7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cc63f0ab8d711bb7"} 21:08:09 [21:08:09.741] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.743] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:37cdf3409006ef2b epochNumber=1 {"provingJobId":"1:BASE_PARITY:37cdf3409006ef2b"} 21:08:09 [21:08:09.744] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testY2AcQv/1 with map size 1073741824 21:08:09 [21:08:09.749] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.778] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a3cd23e39962e6ec epochNumber=1 {"provingJobId":"1:BASE_PARITY:a3cd23e39962e6ec"} 21:08:09 [21:08:09.786] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:37cdf3409006ef2b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:37cdf3409006ef2b"} 21:08:09 [21:08:09.802] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.803] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1fe633efff1eec50 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1fe633efff1eec50"} 21:08:09 [21:08:09.804] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4HHiyc/1 with map size 1073741824 21:08:09 [21:08:09.806] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.828] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:39782c6ffa404d55 epochNumber=1 {"provingJobId":"1:BASE_PARITY:39782c6ffa404d55"} 21:08:09 [21:08:09.835] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1fe633efff1eec50 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:1fe633efff1eec50"} 21:08:09 [21:08:09.849] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.850] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d030ebd72cc4982a epochNumber=1 {"provingJobId":"1:BASE_PARITY:d030ebd72cc4982a"} 21:08:09 [21:08:09.851] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testc7ffca/1 with map size 1073741824 21:08:09 [21:08:09.855] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.881] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b0c0be460d9296c2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b0c0be460d9296c2"} 21:08:09 [21:08:09.888] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d030ebd72cc4982a type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:d030ebd72cc4982a"} 21:08:09 [21:08:09.893] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.894] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:08e7e4af3d7ac88f epochNumber=1 {"provingJobId":"1:BASE_PARITY:08e7e4af3d7ac88f"} 21:08:09 [21:08:09.896] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJqUd8v/1 with map size 1073741824 21:08:09 [21:08:09.898] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.920] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:08e7e4af3d7ac88f {"provingJobId":"1:BASE_PARITY:08e7e4af3d7ac88f"} 21:08:09 [21:08:09.921] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:08e7e4af3d7ac88f type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:08e7e4af3d7ac88f"} 21:08:09 [21:08:09.930] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea99c0a286bde8b9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea99c0a286bde8b9"} 21:08:09 [21:08:09.936] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:08e7e4af3d7ac88f has already been completed {"provingJobId":"1:BASE_PARITY:08e7e4af3d7ac88f"} 21:08:09 [21:08:09.941] INFO: prover-client:proving-broker Proving Broker started 21:08:09 [21:08:09.942] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:89074dd5b3657c5d epochNumber=1 {"provingJobId":"1:BASE_PARITY:89074dd5b3657c5d"} 21:08:09 [21:08:09.944] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrZsEJ3/1 with map size 1073741824 21:08:09 [21:08:09.946] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:09 [21:08:09.970] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:73ebfb2a56cf2b7a epochNumber=2 {"provingJobId":"1:BASE_PARITY:73ebfb2a56cf2b7a"} 21:08:09 [21:08:09.973] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrZsEJ3/2 with map size 1073741824 21:08:09 [21:08:09.978] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:11 [21:08:11.051] INFO: prover-client:proving-broker Proving Broker started 21:08:11 [21:08:11.057] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:89074dd5b3657c5d settled=false {"provingJobId":"1:BASE_PARITY:89074dd5b3657c5d","status":"pending"} 21:08:11 [21:08:11.059] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:73ebfb2a56cf2b7a settled=false {"provingJobId":"1:BASE_PARITY:73ebfb2a56cf2b7a","status":"pending"} 21:08:11 [21:08:11.060] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:89074dd5b3657c5d type=BASE_PARITY startedAt=1750194489999 older agent has taken job {"provingJobId":"1:BASE_PARITY:89074dd5b3657c5d"} 21:08:11 [21:08:11.060] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:89074dd5b3657c5d type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:89074dd5b3657c5d"} 21:08:11 [21:08:11.064] INFO: prover-client:proving-broker Proving Broker started 21:08:11 [21:08:11.065] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d1fa8becfd23d2f6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d1fa8becfd23d2f6"} 21:08:11 [21:08:11.065] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVRQARf/1 with map size 1073741824 21:08:11 [21:08:11.067] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:11 [21:08:11.086] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9e47fe0ac897fa1c epochNumber=2 {"provingJobId":"1:BASE_PARITY:9e47fe0ac897fa1c"} 21:08:11 [21:08:11.088] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVRQARf/2 with map size 1073741824 21:08:11 [21:08:11.091] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:12 [21:08:12.108] INFO: prover-client:proving-broker Proving Broker started 21:08:12 [21:08:12.110] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d1fa8becfd23d2f6 settled=false {"provingJobId":"1:BASE_PARITY:d1fa8becfd23d2f6","status":"pending"} 21:08:12 [21:08:12.111] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9e47fe0ac897fa1c settled=false {"provingJobId":"1:BASE_PARITY:9e47fe0ac897fa1c","status":"pending"} 21:08:12 [21:08:12.112] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d1fa8becfd23d2f6 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:d1fa8becfd23d2f6"} 21:08:12 [21:08:12.116] INFO: prover-client:proving-broker Proving Broker started 21:08:12 [21:08:12.117] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:350744a22d6970c2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:350744a22d6970c2"} 21:08:12 [21:08:12.119] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjVRRPe/1 with map size 1073741824 21:08:12 [21:08:12.122] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:12 [21:08:12.144] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3c04eed77d35456f epochNumber=2 {"provingJobId":"1:BASE_PARITY:3c04eed77d35456f"} 21:08:12 [21:08:12.145] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjVRRPe/2 with map size 1073741824 21:08:12 [21:08:12.151] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.178] INFO: prover-client:proving-broker Proving Broker started 21:08:13 [21:08:13.181] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:350744a22d6970c2 settled=false {"provingJobId":"1:BASE_PARITY:350744a22d6970c2","status":"pending"} 21:08:13 [21:08:13.185] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3c04eed77d35456f settled=false {"provingJobId":"1:BASE_PARITY:3c04eed77d35456f","status":"pending"} 21:08:13 [21:08:13.187] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:350744a22d6970c2 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:350744a22d6970c2"} 21:08:13 [21:08:13.188] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:350744a22d6970c2 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:350744a22d6970c2"} 21:08:13 [21:08:13.197] INFO: prover-client:proving-broker Proving Broker started 21:08:13 [21:08:13.198] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:401e23379d6eecf9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:401e23379d6eecf9"} 21:08:13 [21:08:13.199] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDWJBqp/1 with map size 1073741824 21:08:13 [21:08:13.203] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.222] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:682351984ddea34c epochNumber=2 {"provingJobId":"1:BASE_PARITY:682351984ddea34c"} 21:08:13 [21:08:13.223] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDWJBqp/2 with map size 1073741824 21:08:13 [21:08:13.227] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.248] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:401e23379d6eecf9 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:401e23379d6eecf9"} 21:08:13 [21:08:13.256] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:682351984ddea34c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:682351984ddea34c"} 21:08:13 [21:08:13.276] INFO: prover-client:proving-broker Proving Broker started 21:08:13 [21:08:13.277] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d80d7fcfc339eb76 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d80d7fcfc339eb76"} 21:08:13 [21:08:13.279] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test37Tqkm/1 with map size 1073741824 21:08:13 [21:08:13.281] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.298] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d817d0093bbba960 epochNumber=2 {"provingJobId":"1:BASE_PARITY:d817d0093bbba960"} 21:08:13 [21:08:13.299] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test37Tqkm/2 with map size 1073741824 21:08:13 [21:08:13.302] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.319] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d80d7fcfc339eb76 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d80d7fcfc339eb76"} 21:08:13 [21:08:13.320] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d80d7fcfc339eb76 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d80d7fcfc339eb76"} 21:08:13 [21:08:13.328] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d817d0093bbba960 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:d817d0093bbba960"} 21:08:13 [21:08:13.338] INFO: prover-client:proving-broker Proving Broker started 21:08:13 [21:08:13.339] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:dda512cf74d8f5b2 err=test error {"provingJoId":"1:BASE_PARITY:dda512cf74d8f5b2"} 21:08:13 [21:08:13.344] INFO: prover-client:proving-broker Proving Broker started 21:08:13 [21:08:13.346] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2ad5f888ee8afba4 not found {"provingJobId":"1:BASE_PARITY:2ad5f888ee8afba4"} 21:08:13 [21:08:13.350] INFO: prover-client:proving-broker Proving Broker started 21:08:13 [21:08:13.350] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:01bc5e02d27b28c7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:01bc5e02d27b28c7"} 21:08:13 [21:08:13.352] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQXKFEX/1 with map size 1073741824 21:08:13 [21:08:13.354] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.382] INFO: prover-client:proving-broker Proving Broker started 21:08:13 [21:08:13.383] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fe68ef4a46571e7c epochNumber=1 {"provingJobId":"1:BASE_PARITY:fe68ef4a46571e7c"} 21:08:13 [21:08:13.384] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWWnHwF/1 with map size 1073741824 21:08:13 [21:08:13.389] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.511] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:fe68ef4a46571e7c timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:fe68ef4a46571e7c"} 21:08:13 [21:08:13.518] INFO: prover-client:proving-broker Proving Broker started 21:08:13 [21:08:13.519] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0d5ff4925566fd94 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0d5ff4925566fd94"} 21:08:13 [21:08:13.520] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJ4LxgA/1 with map size 1073741824 21:08:13 [21:08:13.523] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.647] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0d5ff4925566fd94 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:0d5ff4925566fd94"} 21:08:13 [21:08:13.651] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c7fa22aaf9bd08a6 epochNumber=10 {"provingJobId":"1:BASE_PARITY:c7fa22aaf9bd08a6"} 21:08:13 [21:08:13.653] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testJ4LxgA/10 with map size 1073741824 21:08:13 [21:08:13.655] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.698] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:13 [21:08:13.698] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 21:08:13 [21:08:13.813] INFO: prover-client:proving-broker Proving Broker started 21:08:13 [21:08:13.814] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aedb8c7ce79acd88 epochNumber=1 {"provingJobId":"1:BASE_PARITY:aedb8c7ce79acd88"} 21:08:13 [21:08:13.815] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXw9gkK/1 with map size 1073741824 21:08:13 [21:08:13.819] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:13 [21:08:13.995] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:aedb8c7ce79acd88 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:aedb8c7ce79acd88"} 21:08:14 [21:08:14.050] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.051] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:48f66556a7b6fe4f epochNumber=1 {"provingJobId":"1:BASE_PARITY:48f66556a7b6fe4f"} 21:08:14 [21:08:14.053] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYWRSON/1 with map size 1073741824 21:08:14 [21:08:14.061] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.083] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:48f66556a7b6fe4f type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:48f66556a7b6fe4f"} 21:08:14 [21:08:14.088] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.090] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2e0c9d1e91d0dabc epochNumber=1 {"provingJobId":"1:BASE_PARITY:2e0c9d1e91d0dabc"} 21:08:14 [21:08:14.091] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHS0mIs/1 with map size 1073741824 21:08:14 [21:08:14.094] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.115] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2e0c9d1e91d0dabc type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:2e0c9d1e91d0dabc"} 21:08:14 [21:08:14.116] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2e0c9d1e91d0dabc type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:2e0c9d1e91d0dabc"} 21:08:14 [21:08:14.127] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.128] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9067827a0b6717f2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9067827a0b6717f2"} 21:08:14 [21:08:14.129] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUJWnB9/1 with map size 1073741824 21:08:14 [21:08:14.131] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.151] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9067827a0b6717f2 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9067827a0b6717f2"} 21:08:14 [21:08:14.170] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.171] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:474f128b7e306371 epochNumber=1 {"provingJobId":"1:BASE_PARITY:474f128b7e306371"} 21:08:14 [21:08:14.174] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxaYK6e/1 with map size 1073741824 21:08:14 [21:08:14.180] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.200] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:474f128b7e306371 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:474f128b7e306371"} 21:08:14 [21:08:14.202] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ad593299d10ef767 epochNumber=3 {"provingJobId":"1:BASE_PARITY:ad593299d10ef767"} 21:08:14 [21:08:14.203] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testxaYK6e/3 with map size 1073741824 21:08:14 [21:08:14.206] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.248] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:14 [21:08:14.248] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 21:08:14 [21:08:14.251] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:474f128b7e306371 err=test error {"provingJoId":"1:BASE_PARITY:474f128b7e306371"} 21:08:14 [21:08:14.261] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoWLlWs/1 with map size 1073741824 21:08:14 [21:08:14.267] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.291] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testoWLlWs/2 with map size 1073741824 21:08:14 [21:08:14.295] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.317] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.320] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:806097ceb7c79e7d settled=false {"provingJobId":"1:BASE_PARITY:806097ceb7c79e7d","status":"pending"} 21:08:14 [21:08:14.323] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a9aa0642d416e0eb settled=false {"provingJobId":"1:BASE_PARITY:a9aa0642d416e0eb","status":"pending"} 21:08:14 [21:08:14.330] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testea0Z5c/1 with map size 1073741824 21:08:14 [21:08:14.333] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.352] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testea0Z5c/2 with map size 1073741824 21:08:14 [21:08:14.355] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.398] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.402] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:71b9eb418719653c settled=true {"provingJobId":"1:BASE_PARITY:71b9eb418719653c","status":"fulfilled"} 21:08:14 [21:08:14.404] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:ceaa8e8f9306ccc6 settled=true {"provingJobId":"2:BASE_PARITY:ceaa8e8f9306ccc6","status":"fulfilled"} 21:08:14 [21:08:14.411] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVuYwyy/1 with map size 1073741824 21:08:14 [21:08:14.413] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.436] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVuYwyy/2 with map size 1073741824 21:08:14 [21:08:14.439] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.455] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.458] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:45d78ebc122fcbde settled=true {"provingJobId":"1:BASE_PARITY:45d78ebc122fcbde","status":"fulfilled"} 21:08:14 [21:08:14.464] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:54efc80b11acc00d settled=false {"provingJobId":"1:BASE_PARITY:54efc80b11acc00d","status":"pending"} 21:08:14 [21:08:14.470] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.471] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1a08cbad4d8c7028 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1a08cbad4d8c7028"} 21:08:14 [21:08:14.471] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcTaeg1/1 with map size 1073741824 21:08:14 [21:08:14.474] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.499] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.500] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7d448d2cb76b9ce1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7d448d2cb76b9ce1"} 21:08:14 [21:08:14.501] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:7d448d2cb76b9ce1: Error: db error: Error: db error 21:08:14 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:7d448d2cb76b9ce1"} 21:08:14 [21:08:14.505] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.506] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1df91f0c3f53aaea epochNumber=1 {"provingJobId":"1:BASE_PARITY:1df91f0c3f53aaea"} 21:08:14 [21:08:14.508] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test16ppDM/1 with map size 1073741824 21:08:14 [21:08:14.510] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.529] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1df91f0c3f53aaea type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:1df91f0c3f53aaea"} 21:08:14 [21:08:14.529] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1df91f0c3f53aaea type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:1df91f0c3f53aaea"} 21:08:14 [21:08:14.543] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.544] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a9f2e33cc8e4a112 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a9f2e33cc8e4a112"} 21:08:14 [21:08:14.545] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHqEYmz/1 with map size 1073741824 21:08:14 [21:08:14.547] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.566] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a9f2e33cc8e4a112 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a9f2e33cc8e4a112"} 21:08:14 [21:08:14.567] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a9f2e33cc8e4a112 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:a9f2e33cc8e4a112"} 21:08:14 [21:08:14.568] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:a9f2e33cc8e4a112: Error: db error 21:08:14 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:a9f2e33cc8e4a112"} 21:08:14 [21:08:14.573] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.573] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3040a9a9e21dde4c epochNumber=1 {"provingJobId":"1:BASE_PARITY:3040a9a9e21dde4c"} 21:08:14 [21:08:14.575] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnVyKLY/1 with map size 1073741824 21:08:14 [21:08:14.576] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.598] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3040a9a9e21dde4c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3040a9a9e21dde4c"} 21:08:14 [21:08:14.599] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3040a9a9e21dde4c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3040a9a9e21dde4c"} 21:08:14 [21:08:14.611] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.612] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9c59a4c5644c2762 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9c59a4c5644c2762"} 21:08:14 [21:08:14.614] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUDJnfr/1 with map size 1073741824 21:08:14 [21:08:14.617] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.639] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9c59a4c5644c2762 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9c59a4c5644c2762"} 21:08:14 [21:08:14.639] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9c59a4c5644c2762 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9c59a4c5644c2762"} 21:08:14 [21:08:14.640] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:9c59a4c5644c2762 jobErr=test error: Error: db error 21:08:14 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:9c59a4c5644c2762"} 21:08:14 [21:08:14.644] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.645] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:311d844c1836e5fb not found {"provingJobId":"1:BASE_PARITY:311d844c1836e5fb"} 21:08:14 [21:08:14.647] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.648] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:30f73e4a374a4878 err=test error {"provingJoId":"1:BASE_PARITY:30f73e4a374a4878"} 21:08:14 [21:08:14.651] INFO: prover-client:proving-broker Proving Broker started 21:08:14 [21:08:14.662] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:606df70879cb51a3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:606df70879cb51a3"} 21:08:14 [21:08:14.665] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTueBJv/1 with map size 1073741824 21:08:14 [21:08:14.669] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.693] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:606df70879cb51a3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:606df70879cb51a3"} 21:08:14 [21:08:14.693] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:606df70879cb51a3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:606df70879cb51a3"} 21:08:14 [21:08:14.711] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:cb26625f2b807dcd epochNumber=2 {"provingJobId":"2:BASE_PARITY:cb26625f2b807dcd"} 21:08:14 [21:08:14.712] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTueBJv/2 with map size 1073741824 21:08:14 [21:08:14.715] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.734] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:cb26625f2b807dcd type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:cb26625f2b807dcd"} 21:08:14 [21:08:14.735] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:cb26625f2b807dcd type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:cb26625f2b807dcd"} 21:08:14 [21:08:14.803] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:7d40fd1f60e3fe51 epochNumber=3 {"provingJobId":"3:BASE_PARITY:7d40fd1f60e3fe51"} 21:08:14 [21:08:14.804] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testTueBJv/3 with map size 1073741824 21:08:14 [21:08:14.807] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.856] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:14 [21:08:14.857] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 21:08:14 [21:08:14.893] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:7b98cb52eae944a3 epochNumber=4 {"provingJobId":"4:BASE_PARITY:7b98cb52eae944a3"} 21:08:14 [21:08:14.895] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testTueBJv/4 with map size 1073741824 21:08:14 [21:08:14.902] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:14 [21:08:14.935] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:14 [21:08:14.936] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 21:08:14 [21:08:14.983] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:7db2744d3f9ce978 epochNumber=5 {"provingJobId":"5:BASE_PARITY:7db2744d3f9ce978"} 21:08:14 [21:08:14.985] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testTueBJv/5 with map size 1073741824 21:08:14 [21:08:14.991] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:08:15 [21:08:15.013] INFO: prover-client:proving-broker Cleaned up jobs=1 21:08:15 [21:08:15.014] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 21:08:15 PASS src/proving_broker/proving_broker.test.ts (13.56 s) 21:08:15 ProvingBroker 21:08:15 Producer API 21:08:15 ✓ refuses stale jobs (38 ms) 21:08:15 ✓ enqueues jobs (5 ms) 21:08:15 ✓ ignores duplicate jobs (5 ms) 21:08:15 ✓ reports correct status when enqueuing repeat jobs (5 ms) 21:08:15 ✓ throws an error in case of duplicate job IDs (26 ms) 21:08:15 ✓ returns not-found status for non-existing jobs (2 ms) 21:08:15 ✓ cancels jobs in queue (3 ms) 21:08:15 ✓ cancels jobs in-progress (2 ms) 21:08:15 ✓ returns job result if successful (2 ms) 21:08:15 ✓ returns job error if failed (1 ms) 21:08:15 ✓ correctly returns job status for concurrent writes (1 ms) 21:08:15 ✓ refuses stale jobs (63 ms) 21:08:15 ✓ enqueues jobs (50 ms) 21:08:15 ✓ ignores duplicate jobs (44 ms) 21:08:15 ✓ reports correct status when enqueuing repeat jobs (42 ms) 21:08:15 ✓ throws an error in case of duplicate job IDs (42 ms) 21:08:15 ✓ returns not-found status for non-existing jobs (4 ms) 21:08:15 ✓ cancels jobs in queue (42 ms) 21:08:15 ✓ cancels jobs in-progress (51 ms) 21:08:15 ✓ returns job result if successful (42 ms) 21:08:15 ✓ returns job error if failed (44 ms) 21:08:15 ✓ correctly returns job status for concurrent writes (43 ms) 21:08:15 Consumer API 21:08:15 ✓ returns undefined if no jobs are available (2 ms) 21:08:15 ✓ returns jobs in priority order (2 ms) 21:08:15 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 21:08:15 ✓ returns a job if it is in the allowList (2 ms) 21:08:15 ✓ returns the most important job if it is in the allowList (2 ms) 21:08:15 ✓ returns any job if filter is empty (2 ms) 21:08:15 ✓ returns a new job when reporting job success (2 ms) 21:08:15 ✓ returns a new job when reporting permanent error (2 ms) 21:08:15 ✓ returns a new job when reporting retry-able error (2 ms) 21:08:15 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 21:08:15 ✓ returns a new job if job is already in progress elsewhere (1056 ms) 21:08:15 ✓ avoids sending the same job to a new agent after a restart (1004 ms) 21:08:15 ✓ avoids sending a completed job to a new agent after a restart (1004 ms) 21:08:15 ✓ tracks job result if in progress (3 ms) 21:08:15 ✓ tracks job result even if job is in queue (5 ms) 21:08:15 ✓ ignores reported job error if unknown job (2 ms) 21:08:15 ✓ ignores job result if unknown job (2 ms) 21:08:15 ✓ returns undefined if no jobs are available (5 ms) 21:08:15 ✓ returns jobs in priority order (62 ms) 21:08:15 ✓ returns undefined if no jobs are available for the given allowList (27 ms) 21:08:15 ✓ returns a job if it is in the allowList (81 ms) 21:08:15 ✓ returns the most important job if it is in the allowList (66 ms) 21:08:15 ✓ returns any job if filter is empty (83 ms) 21:08:15 ✓ returns a new job when reporting job success (60 ms) 21:08:15 ✓ returns a new job when reporting permanent error (48 ms) 21:08:15 ✓ returns a new job when reporting retry-able error (44 ms) 21:08:15 ✓ returns a new job when reporting progress if current one is cancelled (48 ms) 21:08:15 ✓ returns a new job if job is already in progress elsewhere (1123 ms) 21:08:15 ✓ avoids sending the same job to a new agent after a restart (1052 ms) 21:08:15 ✓ avoids sending a completed job to a new agent after a restart (1080 ms) 21:08:15 ✓ tracks job result if in progress (73 ms) 21:08:15 ✓ tracks job result even if job is in queue (66 ms) 21:08:15 ✓ ignores reported job error if unknown job (3 ms) 21:08:15 ✓ ignores job result if unknown job (8 ms) 21:08:15 Timeouts 21:08:15 ✓ tracks in progress jobs (3 ms) 21:08:15 ✓ re-enqueues jobs that time out (109 ms) 21:08:15 ✓ cancel stale jobs that time out (236 ms) 21:08:15 ✓ keeps the jobs in progress while it is alive (207 ms) 21:08:15 ✓ tracks in progress jobs (33 ms) 21:08:15 ✓ re-enqueues jobs that time out (135 ms) 21:08:15 ✓ cancel stale jobs that time out (294 ms) 21:08:15 ✓ keeps the jobs in progress while it is alive (236 ms) 21:08:15 Retries 21:08:15 ✓ retries jobs (9 ms) 21:08:15 ✓ retries up to a maximum number of times (8 ms) 21:08:15 ✓ passing retry=false does not retry (3 ms) 21:08:15 ✓ does not retry if job is stale (29 ms) 21:08:15 ✓ retries jobs (37 ms) 21:08:15 ✓ retries up to a maximum number of times (38 ms) 21:08:15 ✓ passing retry=false does not retry (42 ms) 21:08:15 ✓ does not retry if job is stale (84 ms) 21:08:15 Database management 21:08:15 ✓ re-enqueues proof requests on start (6 ms) 21:08:15 ✓ restores proof results on start (3 ms) 21:08:15 ✓ only re-enqueues unfinished jobs (2 ms) 21:08:15 ✓ saves job when enqueued (3 ms) 21:08:15 ✓ does not retain job if database fails to save (3 ms) 21:08:15 ✓ saves job result (2 ms) 21:08:15 ✓ saves result even if database fails to save (4 ms) 21:08:15 ✓ saves job error (4 ms) 21:08:15 ✓ saves job error even if database fails to save (8 ms) 21:08:15 ✓ does not save job result if job is unknown (4 ms) 21:08:15 ✓ does not save job error if job is unknown (2 ms) 21:08:15 ✓ cleans up old jobs periodically (269 ms) 21:08:15 ✓ re-enqueues proof requests on start (74 ms) 21:08:15 ✓ restores proof results on start (79 ms) 21:08:15 ✓ only re-enqueues unfinished jobs (59 ms) 21:08:15 ✓ saves job when enqueued (29 ms) 21:08:15 ✓ does not retain job if database fails to save (6 ms) 21:08:15 ✓ saves job result (36 ms) 21:08:15 ✓ saves result even if database fails to save (30 ms) 21:08:15 ✓ saves job error (37 ms) 21:08:15 ✓ saves job error even if database fails to save (34 ms) 21:08:15 ✓ does not save job result if job is unknown (4 ms) 21:08:15 ✓ does not save job error if job is unknown (3 ms) 21:08:15 ✓ cleans up old jobs periodically (417 ms) 21:08:15 21:08:15 Test Suites: 1 passed, 1 total 21:08:15 Tests: 96 passed, 96 total 21:08:15 Snapshots: 0 total 21:08:15 Time: 13.667 s 21:08:15 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 21:08:15 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?