ACI ·

Parent Log: http://ci.aztec-labs.com/3def59c4542ba0a9 Command: 2a84f19e72ab8189 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/69efc9daf98e823f486416900de52fe8a44f25d9 Env: REF_NAME=gh-readonly-queue/next/pr-15187-6e2f55a89973b55c8c3c210b7f3a692b4d3d6473 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Mon Jun 23 16:07:24 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15187_amd64_x1-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 16:07:28 [16:07:28.755] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.761] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0b093b0012bd0eb7 epochNumber=42 {"provingJobId":"1:BASE_PARITY:0b093b0012bd0eb7"} 16:07:28 [16:07:28.764] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:4486df0004abbd38 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4486df0004abbd38"} 16:07:28 [16:07:28.798] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.799] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:110b776dc89c6c1d epochNumber=1 {"provingJobId":"1:BASE_PARITY:110b776dc89c6c1d"} 16:07:28 [16:07:28.800] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b5fdd8422f4f58b7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b5fdd8422f4f58b7"} 16:07:28 [16:07:28.803] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.804] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5f6479b2a59626e4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5f6479b2a59626e4"} 16:07:28 [16:07:28.805] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5f6479b2a59626e4 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:5f6479b2a59626e4"} 16:07:28 [16:07:28.808] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.809] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a229a02b70841e08 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a229a02b70841e08"} 16:07:28 [16:07:28.810] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a229a02b70841e08 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:a229a02b70841e08"} 16:07:28 [16:07:28.811] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a229a02b70841e08 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:a229a02b70841e08"} 16:07:28 [16:07:28.812] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a229a02b70841e08 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:a229a02b70841e08"} 16:07:28 [16:07:28.814] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.815] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6bdb09aed760743f epochNumber=1 {"provingJobId":"1:BASE_PARITY:6bdb09aed760743f"} 16:07:28 [16:07:28.851] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.853] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.853] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1ee8a54be871e832 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1ee8a54be871e832"} 16:07:28 [16:07:28.854] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1ee8a54be871e832 {"provingJobId":"1:BASE_PARITY:1ee8a54be871e832"} 16:07:28 [16:07:28.854] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1ee8a54be871e832 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:1ee8a54be871e832"} 16:07:28 [16:07:28.855] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1ee8a54be871e832 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:1ee8a54be871e832"} 16:07:28 [16:07:28.856] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.857] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eb297155ff6f2f12 epochNumber=1 {"provingJobId":"1:BASE_PARITY:eb297155ff6f2f12"} 16:07:28 [16:07:28.857] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:eb297155ff6f2f12 {"provingJobId":"1:BASE_PARITY:eb297155ff6f2f12"} 16:07:28 [16:07:28.858] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:eb297155ff6f2f12 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:eb297155ff6f2f12"} 16:07:28 [16:07:28.859] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.860] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3bde70da5a5e4c5e epochNumber=1 {"provingJobId":"1:BASE_PARITY:3bde70da5a5e4c5e"} 16:07:28 [16:07:28.860] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3bde70da5a5e4c5e type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3bde70da5a5e4c5e"} 16:07:28 [16:07:28.860] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3bde70da5a5e4c5e type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3bde70da5a5e4c5e"} 16:07:28 [16:07:28.862] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.862] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c80e992c9219adee epochNumber=1 {"provingJobId":"1:BASE_PARITY:c80e992c9219adee"} 16:07:28 [16:07:28.863] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c80e992c9219adee type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c80e992c9219adee"} 16:07:28 [16:07:28.863] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c80e992c9219adee type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:c80e992c9219adee"} 16:07:28 [16:07:28.864] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.864] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8cddda9ba465a495 epochNumber=0 {"provingJobId":"1:BASE_PARITY:8cddda9ba465a495"} 16:07:28 [16:07:28.865] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8cddda9ba465a495 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8cddda9ba465a495"} 16:07:28 [16:07:28.865] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fc6e4920524a2423 epochNumber=0 {"provingJobId":"1:BASE_PARITY:fc6e4920524a2423"} 16:07:28 [16:07:28.865] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8cddda9ba465a495 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8cddda9ba465a495"} 16:07:28 [16:07:28.865] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f70c29f7464b424a epochNumber=0 {"provingJobId":"1:BASE_PARITY:f70c29f7464b424a"} 16:07:28 [16:07:28.868] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.870] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.871] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:35f70fe808d042bb epochNumber=2 {"provingJobId":"1:BASE_PARITY:35f70fe808d042bb"} 16:07:28 [16:07:28.871] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3e6bd98ece53df73 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3e6bd98ece53df73"} 16:07:28 [16:07:28.872] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.873] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:89f9a19644d94765 epochNumber=1 {"provingJobId":"1:BASE_PARITY:89f9a19644d94765"} 16:07:28 [16:07:28.874] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.874] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:75389174b418a6da epochNumber=1 {"provingJobId":"1:BASE_PARITY:75389174b418a6da"} 16:07:28 [16:07:28.874] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:654482b211718cf8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:654482b211718cf8"} 16:07:28 [16:07:28.875] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:25abc2e821c5e382 epochNumber=2 {"provingJobId":"1:BASE_PARITY:25abc2e821c5e382"} 16:07:28 [16:07:28.875] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:402d00e99e548c57 epochNumber=1 {"provingJobId":"1:BASE_PARITY:402d00e99e548c57"} 16:07:28 [16:07:28.876] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.876] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4a3c2273ccfaae8f epochNumber=1 {"provingJobId":"1:BASE_PARITY:4a3c2273ccfaae8f"} 16:07:28 [16:07:28.877] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2ef590e2a95b6a51 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2ef590e2a95b6a51"} 16:07:28 [16:07:28.877] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0cf5abc1cacdfd9b epochNumber=2 {"provingJobId":"1:BASE_PARITY:0cf5abc1cacdfd9b"} 16:07:28 [16:07:28.877] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6b185af36514bf65 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6b185af36514bf65"} 16:07:28 [16:07:28.878] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.879] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0ea1f514100f17a6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0ea1f514100f17a6"} 16:07:28 [16:07:28.879] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:16788ea4f2ef417b epochNumber=1 {"provingJobId":"1:BASE_PARITY:16788ea4f2ef417b"} 16:07:28 [16:07:28.879] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5a209a561da3f715 epochNumber=2 {"provingJobId":"1:BASE_PARITY:5a209a561da3f715"} 16:07:28 [16:07:28.879] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bd8cfe1b572840d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bd8cfe1b572840d6"} 16:07:28 [16:07:28.881] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.881] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6a999c9667d44393 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6a999c9667d44393"} 16:07:28 [16:07:28.881] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a879e7f70363b353 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a879e7f70363b353"} 16:07:28 [16:07:28.882] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6a999c9667d44393 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6a999c9667d44393"} 16:07:28 [16:07:28.883] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.883] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab3b8ec37553a2f6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ab3b8ec37553a2f6"} 16:07:28 [16:07:28.884] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c2c4c762c1fdd26a epochNumber=1 {"provingJobId":"1:BASE_PARITY:c2c4c762c1fdd26a"} 16:07:28 [16:07:28.884] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ab3b8ec37553a2f6 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:ab3b8ec37553a2f6"} 16:07:28 [16:07:28.886] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.887] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:868a939200ad3ab6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:868a939200ad3ab6"} 16:07:28 [16:07:28.887] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4b25a80e03087514 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4b25a80e03087514"} 16:07:28 [16:07:28.888] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:868a939200ad3ab6 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:868a939200ad3ab6"} 16:07:28 [16:07:28.889] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.890] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f1e587bc12cbb27f epochNumber=1 {"provingJobId":"1:BASE_PARITY:f1e587bc12cbb27f"} 16:07:28 [16:07:28.890] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f1e587bc12cbb27f {"provingJobId":"1:BASE_PARITY:f1e587bc12cbb27f"} 16:07:28 [16:07:28.891] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f1e587bc12cbb27f type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:f1e587bc12cbb27f"} 16:07:28 [16:07:28.891] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:39053195351aad5d epochNumber=1 {"provingJobId":"1:BASE_PARITY:39053195351aad5d"} 16:07:28 [16:07:28.892] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f1e587bc12cbb27f has already been completed {"provingJobId":"1:BASE_PARITY:f1e587bc12cbb27f"} 16:07:28 [16:07:28.893] INFO: prover-client:proving-broker Proving Broker started 16:07:28 [16:07:28.893] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:05af6bd0845571a3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:05af6bd0845571a3"} 16:07:28 [16:07:28.893] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1280357f88b238f1 epochNumber=2 {"provingJobId":"1:BASE_PARITY:1280357f88b238f1"} 16:07:29 [16:07:29.945] INFO: prover-client:proving-broker Proving Broker started 16:07:29 [16:07:29.946] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:05af6bd0845571a3 settled=false {"provingJobId":"1:BASE_PARITY:05af6bd0845571a3","status":"pending"} 16:07:29 [16:07:29.947] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1280357f88b238f1 settled=false {"provingJobId":"1:BASE_PARITY:1280357f88b238f1","status":"pending"} 16:07:29 [16:07:29.948] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:05af6bd0845571a3 type=BASE_PARITY startedAt=1750694848894 older agent has taken job {"provingJobId":"1:BASE_PARITY:05af6bd0845571a3"} 16:07:29 [16:07:29.948] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:05af6bd0845571a3 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:05af6bd0845571a3"} 16:07:29 [16:07:29.950] INFO: prover-client:proving-broker Proving Broker started 16:07:29 [16:07:29.950] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:84058ce4d8558be8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:84058ce4d8558be8"} 16:07:29 [16:07:29.951] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d201cca2038dced1 epochNumber=2 {"provingJobId":"1:BASE_PARITY:d201cca2038dced1"} 16:07:30 [16:07:30.951] INFO: prover-client:proving-broker Proving Broker started 16:07:30 [16:07:30.952] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:84058ce4d8558be8 settled=false {"provingJobId":"1:BASE_PARITY:84058ce4d8558be8","status":"pending"} 16:07:30 [16:07:30.952] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d201cca2038dced1 settled=false {"provingJobId":"1:BASE_PARITY:d201cca2038dced1","status":"pending"} 16:07:30 [16:07:30.953] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:84058ce4d8558be8 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:84058ce4d8558be8"} 16:07:30 [16:07:30.955] INFO: prover-client:proving-broker Proving Broker started 16:07:30 [16:07:30.956] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5623f5b69cff0d59 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5623f5b69cff0d59"} 16:07:30 [16:07:30.956] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae367e84d99ce611 epochNumber=2 {"provingJobId":"1:BASE_PARITY:ae367e84d99ce611"} 16:07:31 [16:07:31.957] INFO: prover-client:proving-broker Proving Broker started 16:07:31 [16:07:31.957] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5623f5b69cff0d59 settled=false {"provingJobId":"1:BASE_PARITY:5623f5b69cff0d59","status":"pending"} 16:07:31 [16:07:31.958] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ae367e84d99ce611 settled=false {"provingJobId":"1:BASE_PARITY:ae367e84d99ce611","status":"pending"} 16:07:31 [16:07:31.958] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5623f5b69cff0d59 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5623f5b69cff0d59"} 16:07:31 [16:07:31.959] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5623f5b69cff0d59 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5623f5b69cff0d59"} 16:07:31 [16:07:31.960] INFO: prover-client:proving-broker Proving Broker started 16:07:31 [16:07:31.961] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:edae36d639bad102 epochNumber=1 {"provingJobId":"1:BASE_PARITY:edae36d639bad102"} 16:07:31 [16:07:31.961] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4e68c67bee1125d4 epochNumber=2 {"provingJobId":"1:BASE_PARITY:4e68c67bee1125d4"} 16:07:31 [16:07:31.961] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:edae36d639bad102 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:edae36d639bad102"} 16:07:31 [16:07:31.962] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4e68c67bee1125d4 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4e68c67bee1125d4"} 16:07:31 [16:07:31.963] INFO: prover-client:proving-broker Proving Broker started 16:07:31 [16:07:31.963] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:455d6e6223f50af5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:455d6e6223f50af5"} 16:07:31 [16:07:31.963] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3597be311867d890 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3597be311867d890"} 16:07:31 [16:07:31.964] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:455d6e6223f50af5 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:455d6e6223f50af5"} 16:07:31 [16:07:31.964] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:455d6e6223f50af5 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:455d6e6223f50af5"} 16:07:31 [16:07:31.964] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3597be311867d890 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3597be311867d890"} 16:07:31 [16:07:31.965] INFO: prover-client:proving-broker Proving Broker started 16:07:31 [16:07:31.965] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e76b837fb84f134f err=test error {"provingJoId":"1:BASE_PARITY:e76b837fb84f134f"} 16:07:31 [16:07:31.966] INFO: prover-client:proving-broker Proving Broker started 16:07:31 [16:07:31.967] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a78bb7375d9f681b not found {"provingJobId":"1:BASE_PARITY:a78bb7375d9f681b"} 16:07:31 [16:07:31.968] INFO: prover-client:proving-broker Proving Broker started 16:07:31 [16:07:31.969] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9458908e21e8781e epochNumber=1 {"provingJobId":"1:BASE_PARITY:9458908e21e8781e"} 16:07:31 [16:07:31.970] INFO: prover-client:proving-broker Proving Broker started 16:07:31 [16:07:31.970] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:08f6a5f5c1c7d300 epochNumber=1 {"provingJobId":"1:BASE_PARITY:08f6a5f5c1c7d300"} 16:07:32 [16:07:32.096] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:08f6a5f5c1c7d300 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:08f6a5f5c1c7d300"} 16:07:32 [16:07:32.098] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.098] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:31a597f7e980f849 epochNumber=1 {"provingJobId":"1:BASE_PARITY:31a597f7e980f849"} 16:07:32 [16:07:32.199] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:31a597f7e980f849 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:31a597f7e980f849"} 16:07:32 [16:07:32.206] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f8b23be9b24852fb epochNumber=10 {"provingJobId":"1:BASE_PARITY:f8b23be9b24852fb"} 16:07:32 [16:07:32.225] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:32 [16:07:32.333] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.334] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3da3b7285e0aa9d5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3da3b7285e0aa9d5"} 16:07:32 [16:07:32.488] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3da3b7285e0aa9d5 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:3da3b7285e0aa9d5"} 16:07:32 [16:07:32.539] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.539] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:31f8607e26538354 epochNumber=1 {"provingJobId":"1:BASE_PARITY:31f8607e26538354"} 16:07:32 [16:07:32.540] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:31f8607e26538354 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:31f8607e26538354"} 16:07:32 [16:07:32.541] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.541] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:73c95e6bfacfe783 epochNumber=1 {"provingJobId":"1:BASE_PARITY:73c95e6bfacfe783"} 16:07:32 [16:07:32.542] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:73c95e6bfacfe783 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:73c95e6bfacfe783"} 16:07:32 [16:07:32.542] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:73c95e6bfacfe783 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:73c95e6bfacfe783"} 16:07:32 [16:07:32.543] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.544] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:82b2de6ac6e1d2e6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:82b2de6ac6e1d2e6"} 16:07:32 [16:07:32.544] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:82b2de6ac6e1d2e6 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:82b2de6ac6e1d2e6"} 16:07:32 [16:07:32.545] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.545] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:81ede5242296e8e9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:81ede5242296e8e9"} 16:07:32 [16:07:32.545] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:81ede5242296e8e9 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:81ede5242296e8e9"} 16:07:32 [16:07:32.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f7577acd80e4aaf9 epochNumber=3 {"provingJobId":"1:BASE_PARITY:f7577acd80e4aaf9"} 16:07:32 [16:07:32.570] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:32 [16:07:32.572] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:81ede5242296e8e9 err=test error {"provingJoId":"1:BASE_PARITY:81ede5242296e8e9"} 16:07:32 [16:07:32.574] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.574] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a3663c4dfe321166 settled=false {"provingJobId":"1:BASE_PARITY:a3663c4dfe321166","status":"pending"} 16:07:32 [16:07:32.574] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:37fd8dbe87aa1bd2 settled=false {"provingJobId":"1:BASE_PARITY:37fd8dbe87aa1bd2","status":"pending"} 16:07:32 [16:07:32.577] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.577] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7be0aa2e75914173 settled=true {"provingJobId":"1:BASE_PARITY:7be0aa2e75914173","status":"fulfilled"} 16:07:32 [16:07:32.578] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:1ef14238ce5f8ae8 settled=true {"provingJobId":"2:BASE_PARITY:1ef14238ce5f8ae8","status":"fulfilled"} 16:07:32 [16:07:32.579] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.580] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8967342fa81453cf settled=true {"provingJobId":"1:BASE_PARITY:8967342fa81453cf","status":"fulfilled"} 16:07:32 [16:07:32.580] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:171d8d8bc63925e8 settled=false {"provingJobId":"1:BASE_PARITY:171d8d8bc63925e8","status":"pending"} 16:07:32 [16:07:32.582] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.583] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e9a94346ec856004 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e9a94346ec856004"} 16:07:32 [16:07:32.585] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.586] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9e640b42beddeb74 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9e640b42beddeb74"} 16:07:32 [16:07:32.588] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:9e640b42beddeb74: Error: db error: Error: db error 16:07:32 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:9e640b42beddeb74"} 16:07:32 [16:07:32.589] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.589] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2ea1292aefac8a65 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2ea1292aefac8a65"} 16:07:32 [16:07:32.590] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2ea1292aefac8a65 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2ea1292aefac8a65"} 16:07:32 [16:07:32.590] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2ea1292aefac8a65 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2ea1292aefac8a65"} 16:07:32 [16:07:32.591] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.591] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:57ec19502a691dc8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:57ec19502a691dc8"} 16:07:32 [16:07:32.592] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:57ec19502a691dc8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:57ec19502a691dc8"} 16:07:32 [16:07:32.592] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:57ec19502a691dc8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:57ec19502a691dc8"} 16:07:32 [16:07:32.593] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:57ec19502a691dc8: Error: db error 16:07:32 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:57ec19502a691dc8"} 16:07:32 [16:07:32.594] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.594] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a72824e072549ee2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a72824e072549ee2"} 16:07:32 [16:07:32.594] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a72824e072549ee2 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a72824e072549ee2"} 16:07:32 [16:07:32.595] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a72824e072549ee2 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:a72824e072549ee2"} 16:07:32 [16:07:32.596] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.596] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0bae8d4752f20cad epochNumber=1 {"provingJobId":"1:BASE_PARITY:0bae8d4752f20cad"} 16:07:32 [16:07:32.596] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0bae8d4752f20cad type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0bae8d4752f20cad"} 16:07:32 [16:07:32.596] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0bae8d4752f20cad type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:0bae8d4752f20cad"} 16:07:32 [16:07:32.597] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:0bae8d4752f20cad jobErr=test error: Error: db error 16:07:32 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:0bae8d4752f20cad"} 16:07:32 [16:07:32.598] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.599] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f5058d95ebac799f not found {"provingJobId":"1:BASE_PARITY:f5058d95ebac799f"} 16:07:32 [16:07:32.600] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.600] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:63d5682c0e3d84bc err=test error {"provingJoId":"1:BASE_PARITY:63d5682c0e3d84bc"} 16:07:32 [16:07:32.601] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.611] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:59104415280532fc epochNumber=1 {"provingJobId":"1:BASE_PARITY:59104415280532fc"} 16:07:32 [16:07:32.612] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:59104415280532fc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:59104415280532fc"} 16:07:32 [16:07:32.612] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:59104415280532fc type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:59104415280532fc"} 16:07:32 [16:07:32.622] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:ef0ccfbe6493ffb1 epochNumber=2 {"provingJobId":"2:BASE_PARITY:ef0ccfbe6493ffb1"} 16:07:32 [16:07:32.623] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:ef0ccfbe6493ffb1 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:ef0ccfbe6493ffb1"} 16:07:32 [16:07:32.623] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:ef0ccfbe6493ffb1 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:ef0ccfbe6493ffb1"} 16:07:32 [16:07:32.685] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:747428de8d726bce epochNumber=3 {"provingJobId":"3:BASE_PARITY:747428de8d726bce"} 16:07:32 [16:07:32.703] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:32 [16:07:32.746] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:2050437ec6c6c602 epochNumber=4 {"provingJobId":"4:BASE_PARITY:2050437ec6c6c602"} 16:07:32 [16:07:32.754] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:32 [16:07:32.809] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:24d49bab37ba0af5 epochNumber=5 {"provingJobId":"5:BASE_PARITY:24d49bab37ba0af5"} 16:07:32 [16:07:32.831] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:32 [16:07:32.865] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.866] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dd9a89bd0abe8c66 epochNumber=42 {"provingJobId":"1:BASE_PARITY:dd9a89bd0abe8c66"} 16:07:32 [16:07:32.867] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testMUGwV6/42 with map size 1073741824 16:07:32 [16:07:32.874] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:32 [16:07:32.903] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:0f08c25ceb2ba44f epochNumber=1 {"provingJobId":"1:BASE_PARITY:0f08c25ceb2ba44f"} 16:07:32 [16:07:32.909] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.909] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d68672a8e6ceaaea epochNumber=1 {"provingJobId":"1:BASE_PARITY:d68672a8e6ceaaea"} 16:07:32 [16:07:32.910] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtq1vgA/1 with map size 1073741824 16:07:32 [16:07:32.913] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:32 [16:07:32.935] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4a9ac5794cd07cbf epochNumber=1 {"provingJobId":"1:BASE_PARITY:4a9ac5794cd07cbf"} 16:07:32 [16:07:32.945] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.946] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e18b47a37c57bad7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e18b47a37c57bad7"} 16:07:32 [16:07:32.946] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOmyzwc/1 with map size 1073741824 16:07:32 [16:07:32.949] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:32 [16:07:32.966] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e18b47a37c57bad7 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:e18b47a37c57bad7"} 16:07:32 [16:07:32.970] INFO: prover-client:proving-broker Proving Broker started 16:07:32 [16:07:32.970] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:340e9c95fffe2dfd epochNumber=1 {"provingJobId":"1:BASE_PARITY:340e9c95fffe2dfd"} 16:07:32 [16:07:32.973] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFfVFbq/1 with map size 1073741824 16:07:32 [16:07:32.988] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.010] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:340e9c95fffe2dfd epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:340e9c95fffe2dfd"} 16:07:33 [16:07:33.011] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:340e9c95fffe2dfd type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:340e9c95fffe2dfd"} 16:07:33 [16:07:33.018] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:340e9c95fffe2dfd epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:340e9c95fffe2dfd"} 16:07:33 [16:07:33.024] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.025] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3ac78c2212e1a773 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3ac78c2212e1a773"} 16:07:33 [16:07:33.027] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjG79aG/1 with map size 1073741824 16:07:33 [16:07:33.030] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.055] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.057] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.058] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3c9f2ae11acfc78a epochNumber=1 {"provingJobId":"1:BASE_PARITY:3c9f2ae11acfc78a"} 16:07:33 [16:07:33.058] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test43ngG7/1 with map size 1073741824 16:07:33 [16:07:33.060] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.084] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3c9f2ae11acfc78a {"provingJobId":"1:BASE_PARITY:3c9f2ae11acfc78a"} 16:07:33 [16:07:33.085] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3c9f2ae11acfc78a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3c9f2ae11acfc78a"} 16:07:33 [16:07:33.085] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3c9f2ae11acfc78a type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:3c9f2ae11acfc78a"} 16:07:33 [16:07:33.094] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.094] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6f556c1b3437754e epochNumber=1 {"provingJobId":"1:BASE_PARITY:6f556c1b3437754e"} 16:07:33 [16:07:33.095] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtPNZ3F/1 with map size 1073741824 16:07:33 [16:07:33.099] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.119] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6f556c1b3437754e {"provingJobId":"1:BASE_PARITY:6f556c1b3437754e"} 16:07:33 [16:07:33.120] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6f556c1b3437754e type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:6f556c1b3437754e"} 16:07:33 [16:07:33.129] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.130] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:40dd3ae2db97b9f6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:40dd3ae2db97b9f6"} 16:07:33 [16:07:33.131] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcZcP8b/1 with map size 1073741824 16:07:33 [16:07:33.133] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.151] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:40dd3ae2db97b9f6 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:40dd3ae2db97b9f6"} 16:07:33 [16:07:33.151] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:40dd3ae2db97b9f6 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:40dd3ae2db97b9f6"} 16:07:33 [16:07:33.159] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.160] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3f75be3162481d05 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3f75be3162481d05"} 16:07:33 [16:07:33.161] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYoOuNF/1 with map size 1073741824 16:07:33 [16:07:33.164] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.194] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3f75be3162481d05 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3f75be3162481d05"} 16:07:33 [16:07:33.194] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3f75be3162481d05 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3f75be3162481d05"} 16:07:33 [16:07:33.206] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.206] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7b13dbd2380578c9 epochNumber=0 {"provingJobId":"1:BASE_PARITY:7b13dbd2380578c9"} 16:07:33 [16:07:33.208] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test9ax19p/0 with map size 1073741824 16:07:33 [16:07:33.211] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.233] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7b13dbd2380578c9 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:7b13dbd2380578c9"} 16:07:33 [16:07:33.234] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d72095f68ea6fba2 epochNumber=0 {"provingJobId":"1:BASE_PARITY:d72095f68ea6fba2"} 16:07:33 [16:07:33.234] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7b13dbd2380578c9 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:7b13dbd2380578c9"} 16:07:33 [16:07:33.235] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5a29f790f84cdc41 epochNumber=0 {"provingJobId":"1:BASE_PARITY:5a29f790f84cdc41"} 16:07:33 [16:07:33.250] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.253] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.254] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e2c46f9038d1ca54 epochNumber=2 {"provingJobId":"1:BASE_PARITY:e2c46f9038d1ca54"} 16:07:33 [16:07:33.254] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testsjhBwB/2 with map size 1073741824 16:07:33 [16:07:33.256] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.282] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:668d3ab936d3ec79 epochNumber=1 {"provingJobId":"1:BASE_PARITY:668d3ab936d3ec79"} 16:07:33 [16:07:33.285] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsjhBwB/1 with map size 1073741824 16:07:33 [16:07:33.289] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.312] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.312] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:368365b9e36b71ae epochNumber=1 {"provingJobId":"1:BASE_PARITY:368365b9e36b71ae"} 16:07:33 [16:07:33.313] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdSmrAr/1 with map size 1073741824 16:07:33 [16:07:33.315] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.336] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.336] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d7436839b90e72ed epochNumber=1 {"provingJobId":"1:BASE_PARITY:d7436839b90e72ed"} 16:07:33 [16:07:33.337] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnBFa3b/1 with map size 1073741824 16:07:33 [16:07:33.339] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.356] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1b22cac5e4ef3a8d epochNumber=1 {"provingJobId":"1:BASE_PARITY:1b22cac5e4ef3a8d"} 16:07:33 [16:07:33.362] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:78ee5d032da8333b epochNumber=2 {"provingJobId":"1:BASE_PARITY:78ee5d032da8333b"} 16:07:33 [16:07:33.363] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnBFa3b/2 with map size 1073741824 16:07:33 [16:07:33.367] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.395] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:32ca2185eb49b1e5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:32ca2185eb49b1e5"} 16:07:33 [16:07:33.407] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.408] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1960cea6e91f1c91 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1960cea6e91f1c91"} 16:07:33 [16:07:33.409] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKJXG5r/1 with map size 1073741824 16:07:33 [16:07:33.411] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.427] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:643041483ba763cf epochNumber=1 {"provingJobId":"1:BASE_PARITY:643041483ba763cf"} 16:07:33 [16:07:33.433] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4b108fd42a3c2505 epochNumber=2 {"provingJobId":"1:BASE_PARITY:4b108fd42a3c2505"} 16:07:33 [16:07:33.434] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKJXG5r/2 with map size 1073741824 16:07:33 [16:07:33.437] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.455] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:86bfdf0aea0e0bfe epochNumber=1 {"provingJobId":"1:BASE_PARITY:86bfdf0aea0e0bfe"} 16:07:33 [16:07:33.464] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.464] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:32c2b61fd131e7e1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:32c2b61fd131e7e1"} 16:07:33 [16:07:33.465] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testk5gS4f/1 with map size 1073741824 16:07:33 [16:07:33.466] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.494] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:83984dcd1174af99 epochNumber=1 {"provingJobId":"1:BASE_PARITY:83984dcd1174af99"} 16:07:33 [16:07:33.501] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:333a282b2be2d460 epochNumber=2 {"provingJobId":"1:BASE_PARITY:333a282b2be2d460"} 16:07:33 [16:07:33.502] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testk5gS4f/2 with map size 1073741824 16:07:33 [16:07:33.504] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.520] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cc857e015c54d50a epochNumber=1 {"provingJobId":"1:BASE_PARITY:cc857e015c54d50a"} 16:07:33 [16:07:33.529] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.529] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6a0ae4b31c07f50f epochNumber=1 {"provingJobId":"1:BASE_PARITY:6a0ae4b31c07f50f"} 16:07:33 [16:07:33.530] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFld7hO/1 with map size 1073741824 16:07:33 [16:07:33.531] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a979ec9ef1ad7577 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a979ec9ef1ad7577"} 16:07:33 [16:07:33.552] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6a0ae4b31c07f50f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6a0ae4b31c07f50f"} 16:07:33 [16:07:33.560] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.561] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3fd8fbb7482c21e1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3fd8fbb7482c21e1"} 16:07:33 [16:07:33.561] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testx0I3j1/1 with map size 1073741824 16:07:33 [16:07:33.562] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.587] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b4e4307d4e781735 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b4e4307d4e781735"} 16:07:33 [16:07:33.594] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3fd8fbb7482c21e1 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:3fd8fbb7482c21e1"} 16:07:33 [16:07:33.605] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.606] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a753626c7e55f1d8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a753626c7e55f1d8"} 16:07:33 [16:07:33.607] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQhSciM/1 with map size 1073741824 16:07:33 [16:07:33.609] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.628] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5f8f71f5ed6ffff0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5f8f71f5ed6ffff0"} 16:07:33 [16:07:33.633] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a753626c7e55f1d8 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:a753626c7e55f1d8"} 16:07:33 [16:07:33.637] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.637] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:16905e7fa8f17d62 epochNumber=1 {"provingJobId":"1:BASE_PARITY:16905e7fa8f17d62"} 16:07:33 [16:07:33.638] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlBouaq/1 with map size 1073741824 16:07:33 [16:07:33.639] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.654] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:16905e7fa8f17d62 {"provingJobId":"1:BASE_PARITY:16905e7fa8f17d62"} 16:07:33 [16:07:33.654] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:16905e7fa8f17d62 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:16905e7fa8f17d62"} 16:07:33 [16:07:33.660] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e913835e3529a09c epochNumber=1 {"provingJobId":"1:BASE_PARITY:e913835e3529a09c"} 16:07:33 [16:07:33.667] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:16905e7fa8f17d62 has already been completed {"provingJobId":"1:BASE_PARITY:16905e7fa8f17d62"} 16:07:33 [16:07:33.670] INFO: prover-client:proving-broker Proving Broker started 16:07:33 [16:07:33.670] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2c0c6220749bdd54 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2c0c6220749bdd54"} 16:07:33 [16:07:33.673] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQQbRKD/1 with map size 1073741824 16:07:33 [16:07:33.678] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:33 [16:07:33.699] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2e8f0ded0f07d50f epochNumber=2 {"provingJobId":"1:BASE_PARITY:2e8f0ded0f07d50f"} 16:07:33 [16:07:33.699] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQQbRKD/2 with map size 1073741824 16:07:33 [16:07:33.702] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:34 [16:07:34.772] INFO: prover-client:proving-broker Proving Broker started 16:07:34 [16:07:34.780] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2c0c6220749bdd54 settled=false {"provingJobId":"1:BASE_PARITY:2c0c6220749bdd54","status":"pending"} 16:07:34 [16:07:34.782] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2e8f0ded0f07d50f settled=false {"provingJobId":"1:BASE_PARITY:2e8f0ded0f07d50f","status":"pending"} 16:07:34 [16:07:34.783] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:2c0c6220749bdd54 type=BASE_PARITY startedAt=1750694853718 older agent has taken job {"provingJobId":"1:BASE_PARITY:2c0c6220749bdd54"} 16:07:34 [16:07:34.784] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2c0c6220749bdd54 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:2c0c6220749bdd54"} 16:07:34 [16:07:34.792] INFO: prover-client:proving-broker Proving Broker started 16:07:34 [16:07:34.794] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b3103e683660b089 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b3103e683660b089"} 16:07:34 [16:07:34.795] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHA38Yg/1 with map size 1073741824 16:07:34 [16:07:34.798] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:34 [16:07:34.821] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:425adb921f65be04 epochNumber=2 {"provingJobId":"1:BASE_PARITY:425adb921f65be04"} 16:07:34 [16:07:34.823] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testHA38Yg/2 with map size 1073741824 16:07:34 [16:07:34.825] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:35 [16:07:35.843] INFO: prover-client:proving-broker Proving Broker started 16:07:35 [16:07:35.846] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b3103e683660b089 settled=false {"provingJobId":"1:BASE_PARITY:b3103e683660b089","status":"pending"} 16:07:35 [16:07:35.847] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:425adb921f65be04 settled=false {"provingJobId":"1:BASE_PARITY:425adb921f65be04","status":"pending"} 16:07:35 [16:07:35.848] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b3103e683660b089 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:b3103e683660b089"} 16:07:35 [16:07:35.853] INFO: prover-client:proving-broker Proving Broker started 16:07:35 [16:07:35.854] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9b6e74a1b6677da8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9b6e74a1b6677da8"} 16:07:35 [16:07:35.855] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkYok5w/1 with map size 1073741824 16:07:35 [16:07:35.857] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:35 [16:07:35.882] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3f3e1d09762bbd2b epochNumber=2 {"provingJobId":"1:BASE_PARITY:3f3e1d09762bbd2b"} 16:07:35 [16:07:35.883] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkYok5w/2 with map size 1073741824 16:07:35 [16:07:35.885] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:36 [16:07:36.906] INFO: prover-client:proving-broker Proving Broker started 16:07:36 [16:07:36.910] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9b6e74a1b6677da8 settled=false {"provingJobId":"1:BASE_PARITY:9b6e74a1b6677da8","status":"pending"} 16:07:36 [16:07:36.911] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3f3e1d09762bbd2b settled=false {"provingJobId":"1:BASE_PARITY:3f3e1d09762bbd2b","status":"pending"} 16:07:36 [16:07:36.912] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9b6e74a1b6677da8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9b6e74a1b6677da8"} 16:07:36 [16:07:36.912] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9b6e74a1b6677da8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:9b6e74a1b6677da8"} 16:07:36 [16:07:36.921] INFO: prover-client:proving-broker Proving Broker started 16:07:36 [16:07:36.922] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:307c768050a09b74 epochNumber=1 {"provingJobId":"1:BASE_PARITY:307c768050a09b74"} 16:07:36 [16:07:36.922] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testO4vy7k/1 with map size 1073741824 16:07:36 [16:07:36.925] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:36 [16:07:36.940] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d96daf6a6c96990d epochNumber=2 {"provingJobId":"1:BASE_PARITY:d96daf6a6c96990d"} 16:07:36 [16:07:36.941] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testO4vy7k/2 with map size 1073741824 16:07:36 [16:07:36.943] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:36 [16:07:36.959] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:307c768050a09b74 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:307c768050a09b74"} 16:07:36 [16:07:36.965] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d96daf6a6c96990d type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:d96daf6a6c96990d"} 16:07:36 [16:07:36.982] INFO: prover-client:proving-broker Proving Broker started 16:07:36 [16:07:36.983] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d6c7a49d35d2e25e epochNumber=1 {"provingJobId":"1:BASE_PARITY:d6c7a49d35d2e25e"} 16:07:36 [16:07:36.986] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfxM58G/1 with map size 1073741824 16:07:36 [16:07:36.988] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.009] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ce03fecf63f0792c epochNumber=2 {"provingJobId":"1:BASE_PARITY:ce03fecf63f0792c"} 16:07:37 [16:07:37.010] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfxM58G/2 with map size 1073741824 16:07:37 [16:07:37.014] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.037] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d6c7a49d35d2e25e type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d6c7a49d35d2e25e"} 16:07:37 [16:07:37.037] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d6c7a49d35d2e25e type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d6c7a49d35d2e25e"} 16:07:37 [16:07:37.045] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ce03fecf63f0792c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ce03fecf63f0792c"} 16:07:37 [16:07:37.055] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.056] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4341e43f473fa3f3 err=test error {"provingJoId":"1:BASE_PARITY:4341e43f473fa3f3"} 16:07:37 [16:07:37.060] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.061] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c815d90076a8fef9 not found {"provingJobId":"1:BASE_PARITY:c815d90076a8fef9"} 16:07:37 [16:07:37.067] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.067] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:39a7c57a12a9a6e1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:39a7c57a12a9a6e1"} 16:07:37 [16:07:37.068] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testandot2/1 with map size 1073741824 16:07:37 [16:07:37.070] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.104] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.105] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b14450bb67d5fff0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b14450bb67d5fff0"} 16:07:37 [16:07:37.106] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNXE4ps/1 with map size 1073741824 16:07:37 [16:07:37.108] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.232] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b14450bb67d5fff0 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:b14450bb67d5fff0"} 16:07:37 [16:07:37.242] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.243] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a4d3a88d2b3c5674 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a4d3a88d2b3c5674"} 16:07:37 [16:07:37.243] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXrNpNQ/1 with map size 1073741824 16:07:37 [16:07:37.245] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.369] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a4d3a88d2b3c5674 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:a4d3a88d2b3c5674"} 16:07:37 [16:07:37.377] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dec0e59d2d9f80f1 epochNumber=10 {"provingJobId":"1:BASE_PARITY:dec0e59d2d9f80f1"} 16:07:37 [16:07:37.383] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testXrNpNQ/10 with map size 1073741824 16:07:37 [16:07:37.387] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.422] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:37 [16:07:37.422] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:07:37 [16:07:37.540] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.541] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a3098ba0cac21d03 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a3098ba0cac21d03"} 16:07:37 [16:07:37.541] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test27Q8zz/1 with map size 1073741824 16:07:37 [16:07:37.545] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.719] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a3098ba0cac21d03 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:a3098ba0cac21d03"} 16:07:37 [16:07:37.774] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.775] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a9fb2b4936723a6d epochNumber=1 {"provingJobId":"1:BASE_PARITY:a9fb2b4936723a6d"} 16:07:37 [16:07:37.776] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLaYi0H/1 with map size 1073741824 16:07:37 [16:07:37.779] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.803] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a9fb2b4936723a6d type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:a9fb2b4936723a6d"} 16:07:37 [16:07:37.807] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.808] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:206793426ba124a3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:206793426ba124a3"} 16:07:37 [16:07:37.809] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlqqNTO/1 with map size 1073741824 16:07:37 [16:07:37.813] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.832] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:206793426ba124a3 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:206793426ba124a3"} 16:07:37 [16:07:37.833] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:206793426ba124a3 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:206793426ba124a3"} 16:07:37 [16:07:37.845] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.846] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2810389475595e75 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2810389475595e75"} 16:07:37 [16:07:37.846] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxdnB9T/1 with map size 1073741824 16:07:37 [16:07:37.848] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.866] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2810389475595e75 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:2810389475595e75"} 16:07:37 [16:07:37.895] INFO: prover-client:proving-broker Proving Broker started 16:07:37 [16:07:37.896] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d8bd0024cadc1418 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d8bd0024cadc1418"} 16:07:37 [16:07:37.897] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPn1gjf/1 with map size 1073741824 16:07:37 [16:07:37.900] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.918] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d8bd0024cadc1418 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:d8bd0024cadc1418"} 16:07:37 [16:07:37.920] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:adede8827d27065f epochNumber=3 {"provingJobId":"1:BASE_PARITY:adede8827d27065f"} 16:07:37 [16:07:37.924] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testPn1gjf/3 with map size 1073741824 16:07:37 [16:07:37.931] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:37 [16:07:37.978] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:37 [16:07:37.978] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:07:38 [16:07:38.001] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d8bd0024cadc1418 err=test error {"provingJoId":"1:BASE_PARITY:d8bd0024cadc1418"} 16:07:38 [16:07:38.006] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmh1IX2/1 with map size 1073741824 16:07:38 [16:07:38.010] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.036] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testmh1IX2/2 with map size 1073741824 16:07:38 [16:07:38.038] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.059] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.063] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ff9ad8ec2866dc87 settled=false {"provingJobId":"1:BASE_PARITY:ff9ad8ec2866dc87","status":"pending"} 16:07:38 [16:07:38.065] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f4070b9da5d189e6 settled=false {"provingJobId":"1:BASE_PARITY:f4070b9da5d189e6","status":"pending"} 16:07:38 [16:07:38.074] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testm5kSxP/1 with map size 1073741824 16:07:38 [16:07:38.079] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.102] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testm5kSxP/2 with map size 1073741824 16:07:38 [16:07:38.107] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.145] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.149] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:53dd0d07ee566c56 settled=true {"provingJobId":"1:BASE_PARITY:53dd0d07ee566c56","status":"fulfilled"} 16:07:38 [16:07:38.151] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:cc82a1f36935e975 settled=true {"provingJobId":"2:BASE_PARITY:cc82a1f36935e975","status":"fulfilled"} 16:07:38 [16:07:38.160] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testD2cfAX/1 with map size 1073741824 16:07:38 [16:07:38.162] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.196] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testD2cfAX/2 with map size 1073741824 16:07:38 [16:07:38.199] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.221] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.223] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:862b93e9c00b3843 settled=true {"provingJobId":"1:BASE_PARITY:862b93e9c00b3843","status":"fulfilled"} 16:07:38 [16:07:38.226] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d6f8faedab65482a settled=false {"provingJobId":"1:BASE_PARITY:d6f8faedab65482a","status":"pending"} 16:07:38 [16:07:38.232] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.233] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5cb76eafc18ce256 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5cb76eafc18ce256"} 16:07:38 [16:07:38.234] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOFNgeb/1 with map size 1073741824 16:07:38 [16:07:38.237] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.261] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.263] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dcc15ddc6fd3a220 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dcc15ddc6fd3a220"} 16:07:38 [16:07:38.264] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:dcc15ddc6fd3a220: Error: db error: Error: db error 16:07:38 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:dcc15ddc6fd3a220"} 16:07:38 [16:07:38.269] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.269] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:82b7bb475e8de1fe epochNumber=1 {"provingJobId":"1:BASE_PARITY:82b7bb475e8de1fe"} 16:07:38 [16:07:38.270] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testv0QcUi/1 with map size 1073741824 16:07:38 [16:07:38.280] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.303] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:82b7bb475e8de1fe type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:82b7bb475e8de1fe"} 16:07:38 [16:07:38.304] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:82b7bb475e8de1fe type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:82b7bb475e8de1fe"} 16:07:38 [16:07:38.315] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.316] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b7b85479b377b789 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b7b85479b377b789"} 16:07:38 [16:07:38.316] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testh0wqsn/1 with map size 1073741824 16:07:38 [16:07:38.320] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.342] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b7b85479b377b789 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b7b85479b377b789"} 16:07:38 [16:07:38.342] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b7b85479b377b789 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b7b85479b377b789"} 16:07:38 [16:07:38.343] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:b7b85479b377b789: Error: db error 16:07:38 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:b7b85479b377b789"} 16:07:38 [16:07:38.350] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.351] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:12129683175ca8d9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:12129683175ca8d9"} 16:07:38 [16:07:38.353] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqKJJxd/1 with map size 1073741824 16:07:38 [16:07:38.355] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.377] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:12129683175ca8d9 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:12129683175ca8d9"} 16:07:38 [16:07:38.378] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:12129683175ca8d9 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:12129683175ca8d9"} 16:07:38 [16:07:38.395] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.397] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b74c890f5b488fee epochNumber=1 {"provingJobId":"1:BASE_PARITY:b74c890f5b488fee"} 16:07:38 [16:07:38.398] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testG7IPbh/1 with map size 1073741824 16:07:38 [16:07:38.401] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.419] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b74c890f5b488fee type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b74c890f5b488fee"} 16:07:38 [16:07:38.419] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b74c890f5b488fee type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b74c890f5b488fee"} 16:07:38 [16:07:38.420] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:b74c890f5b488fee jobErr=test error: Error: db error 16:07:38 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:b74c890f5b488fee"} 16:07:38 [16:07:38.425] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.426] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:072cfcaa290d41e1 not found {"provingJobId":"1:BASE_PARITY:072cfcaa290d41e1"} 16:07:38 [16:07:38.430] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.431] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b21a95a43be5fbb3 err=test error {"provingJoId":"1:BASE_PARITY:b21a95a43be5fbb3"} 16:07:38 [16:07:38.437] INFO: prover-client:proving-broker Proving Broker started 16:07:38 [16:07:38.449] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8c5bb1f5b1ea4e8e epochNumber=1 {"provingJobId":"1:BASE_PARITY:8c5bb1f5b1ea4e8e"} 16:07:38 [16:07:38.450] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnTCkEb/1 with map size 1073741824 16:07:38 [16:07:38.452] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.466] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8c5bb1f5b1ea4e8e type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8c5bb1f5b1ea4e8e"} 16:07:38 [16:07:38.467] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8c5bb1f5b1ea4e8e type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8c5bb1f5b1ea4e8e"} 16:07:38 [16:07:38.485] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:7a582e0260b9fb81 epochNumber=2 {"provingJobId":"2:BASE_PARITY:7a582e0260b9fb81"} 16:07:38 [16:07:38.486] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnTCkEb/2 with map size 1073741824 16:07:38 [16:07:38.490] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.508] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:7a582e0260b9fb81 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:7a582e0260b9fb81"} 16:07:38 [16:07:38.509] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:7a582e0260b9fb81 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:7a582e0260b9fb81"} 16:07:38 [16:07:38.577] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:13de2385692999b2 epochNumber=3 {"provingJobId":"3:BASE_PARITY:13de2385692999b2"} 16:07:38 [16:07:38.581] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testnTCkEb/3 with map size 1073741824 16:07:38 [16:07:38.583] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.615] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:38 [16:07:38.616] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:07:38 [16:07:38.665] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:ad66021483e7643a epochNumber=4 {"provingJobId":"4:BASE_PARITY:ad66021483e7643a"} 16:07:38 [16:07:38.668] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testnTCkEb/4 with map size 1073741824 16:07:38 [16:07:38.672] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.719] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:38 [16:07:38.720] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 16:07:38 [16:07:38.755] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:93fb34ee80b51958 epochNumber=5 {"provingJobId":"5:BASE_PARITY:93fb34ee80b51958"} 16:07:38 [16:07:38.757] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testnTCkEb/5 with map size 1073741824 16:07:38 [16:07:38.761] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:07:38 [16:07:38.800] INFO: prover-client:proving-broker Cleaned up jobs=1 16:07:38 [16:07:38.801] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 16:07:38 PASS src/proving_broker/proving_broker.test.ts (13.63 s) 16:07:38 ProvingBroker 16:07:38 Producer API 16:07:38 ✓ refuses stale jobs (44 ms) 16:07:38 ✓ enqueues jobs (5 ms) 16:07:38 ✓ ignores duplicate jobs (5 ms) 16:07:38 ✓ reports correct status when enqueuing repeat jobs (6 ms) 16:07:38 ✓ throws an error in case of duplicate job IDs (37 ms) 16:07:38 ✓ returns not-found status for non-existing jobs (1 ms) 16:07:38 ✓ cancels jobs in queue (2 ms) 16:07:38 ✓ cancels jobs in-progress (2 ms) 16:07:38 ✓ returns job result if successful (3 ms) 16:07:38 ✓ returns job error if failed (2 ms) 16:07:38 ✓ correctly returns job status for concurrent writes (3 ms) 16:07:38 ✓ refuses stale jobs (44 ms) 16:07:38 ✓ enqueues jobs (36 ms) 16:07:38 ✓ ignores duplicate jobs (25 ms) 16:07:38 ✓ reports correct status when enqueuing repeat jobs (52 ms) 16:07:38 ✓ throws an error in case of duplicate job IDs (32 ms) 16:07:38 ✓ returns not-found status for non-existing jobs (3 ms) 16:07:38 ✓ cancels jobs in queue (36 ms) 16:07:38 ✓ cancels jobs in-progress (34 ms) 16:07:38 ✓ returns job result if successful (30 ms) 16:07:38 ✓ returns job error if failed (46 ms) 16:07:38 ✓ correctly returns job status for concurrent writes (45 ms) 16:07:38 Consumer API 16:07:38 ✓ returns undefined if no jobs are available (2 ms) 16:07:38 ✓ returns jobs in priority order (2 ms) 16:07:38 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 16:07:38 ✓ returns a job if it is in the allowList (2 ms) 16:07:38 ✓ returns the most important job if it is in the allowList (2 ms) 16:07:38 ✓ returns any job if filter is empty (2 ms) 16:07:38 ✓ returns a new job when reporting job success (2 ms) 16:07:38 ✓ returns a new job when reporting permanent error (1 ms) 16:07:38 ✓ returns a new job when reporting retry-able error (4 ms) 16:07:38 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 16:07:38 ✓ returns a new job if job is already in progress elsewhere (1057 ms) 16:07:38 ✓ avoids sending the same job to a new agent after a restart (1005 ms) 16:07:38 ✓ avoids sending a completed job to a new agent after a restart (1004 ms) 16:07:38 ✓ tracks job result if in progress (2 ms) 16:07:38 ✓ tracks job result even if job is in queue (1 ms) 16:07:38 ✓ ignores reported job error if unknown job (1 ms) 16:07:38 ✓ ignores job result if unknown job (1 ms) 16:07:38 ✓ returns undefined if no jobs are available (2 ms) 16:07:38 ✓ returns jobs in priority order (59 ms) 16:07:38 ✓ returns undefined if no jobs are available for the given allowList (23 ms) 16:07:38 ✓ returns a job if it is in the allowList (70 ms) 16:07:38 ✓ returns the most important job if it is in the allowList (57 ms) 16:07:38 ✓ returns any job if filter is empty (65 ms) 16:07:38 ✓ returns a new job when reporting job success (31 ms) 16:07:38 ✓ returns a new job when reporting permanent error (45 ms) 16:07:38 ✓ returns a new job when reporting retry-able error (32 ms) 16:07:38 ✓ returns a new job when reporting progress if current one is cancelled (33 ms) 16:07:38 ✓ returns a new job if job is already in progress elsewhere (1121 ms) 16:07:38 ✓ avoids sending the same job to a new agent after a restart (1061 ms) 16:07:38 ✓ avoids sending a completed job to a new agent after a restart (1067 ms) 16:07:38 ✓ tracks job result if in progress (54 ms) 16:07:38 ✓ tracks job result even if job is in queue (78 ms) 16:07:38 ✓ ignores reported job error if unknown job (4 ms) 16:07:38 ✓ ignores job result if unknown job (4 ms) 16:07:38 Timeouts 16:07:38 ✓ tracks in progress jobs (1 ms) 16:07:38 ✓ re-enqueues jobs that time out (127 ms) 16:07:38 ✓ cancel stale jobs that time out (236 ms) 16:07:38 ✓ keeps the jobs in progress while it is alive (205 ms) 16:07:38 ✓ tracks in progress jobs (38 ms) 16:07:38 ✓ re-enqueues jobs that time out (138 ms) 16:07:38 ✓ cancel stale jobs that time out (296 ms) 16:07:38 ✓ keeps the jobs in progress while it is alive (234 ms) 16:07:38 Retries 16:07:38 ✓ retries jobs (3 ms) 16:07:38 ✓ retries up to a maximum number of times (2 ms) 16:07:38 ✓ passing retry=false does not retry (1 ms) 16:07:38 ✓ does not retry if job is stale (28 ms) 16:07:38 ✓ retries jobs (34 ms) 16:07:38 ✓ retries up to a maximum number of times (37 ms) 16:07:38 ✓ passing retry=false does not retry (35 ms) 16:07:38 ✓ does not retry if job is stale (126 ms) 16:07:38 Database management 16:07:38 ✓ re-enqueues proof requests on start (3 ms) 16:07:38 ✓ restores proof results on start (3 ms) 16:07:38 ✓ only re-enqueues unfinished jobs (2 ms) 16:07:38 ✓ saves job when enqueued (3 ms) 16:07:38 ✓ does not retain job if database fails to save (3 ms) 16:07:38 ✓ saves job result (2 ms) 16:07:38 ✓ saves result even if database fails to save (2 ms) 16:07:38 ✓ saves job error (2 ms) 16:07:38 ✓ saves job error even if database fails to save (2 ms) 16:07:38 ✓ does not save job result if job is unknown (1 ms) 16:07:38 ✓ does not save job error if job is unknown (2 ms) 16:07:38 ✓ cleans up old jobs periodically (260 ms) 16:07:38 ✓ re-enqueues proof requests on start (66 ms) 16:07:38 ✓ restores proof results on start (84 ms) 16:07:38 ✓ only re-enqueues unfinished jobs (74 ms) 16:07:38 ✓ saves job when enqueued (28 ms) 16:07:38 ✓ does not retain job if database fails to save (7 ms) 16:07:38 ✓ saves job result (47 ms) 16:07:38 ✓ saves result even if database fails to save (34 ms) 16:07:38 ✓ saves job error (43 ms) 16:07:38 ✓ saves job error even if database fails to save (30 ms) 16:07:38 ✓ does not save job result if job is unknown (5 ms) 16:07:38 ✓ does not save job error if job is unknown (4 ms) 16:07:38 ✓ cleans up old jobs periodically (411 ms) 16:07:38 16:07:38 Test Suites: 1 passed, 1 total 16:07:38 Tests: 96 passed, 96 total 16:07:38 Snapshots: 0 total 16:07:38 Time: 13.728 s 16:07:38 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 16:07:38 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?