ACI ·

Parent Log: http://ci.aztec-labs.com/b122f2324120c710 Command: f682b45a820409ef yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/75d792847d8434a0c504e7adf5c102a913065272 Env: REF_NAME=gh-readonly-queue/next/pr-15015-cf8be0f9e81e248048560619de041e90d9d6990a CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Fri Jun 13 08:35:10 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15015_amd64_x2-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 08:35:14 [08:35:14.358] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.366] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6c39962f4886c92b epochNumber=42 {"provingJobId":"1:BASE_PARITY:6c39962f4886c92b"} 08:35:14 [08:35:14.370] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:529c30ddaf57f785 epochNumber=1 {"provingJobId":"1:BASE_PARITY:529c30ddaf57f785"} 08:35:14 [08:35:14.423] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.426] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dd7ffbee60d71afa epochNumber=1 {"provingJobId":"1:BASE_PARITY:dd7ffbee60d71afa"} 08:35:14 [08:35:14.430] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4d3634c1a91dcb99 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4d3634c1a91dcb99"} 08:35:14 [08:35:14.432] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.434] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3755542d220e6f35 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3755542d220e6f35"} 08:35:14 [08:35:14.436] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3755542d220e6f35 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:3755542d220e6f35"} 08:35:14 [08:35:14.440] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.442] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b720dc912b538d15 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b720dc912b538d15"} 08:35:14 [08:35:14.443] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b720dc912b538d15 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b720dc912b538d15"} 08:35:14 [08:35:14.444] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b720dc912b538d15 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b720dc912b538d15"} 08:35:14 [08:35:14.445] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b720dc912b538d15 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b720dc912b538d15"} 08:35:14 [08:35:14.447] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.448] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0a56a502ac394cbb epochNumber=1 {"provingJobId":"1:BASE_PARITY:0a56a502ac394cbb"} 08:35:14 [08:35:14.481] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.485] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.485] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d638705d48916d7b epochNumber=1 {"provingJobId":"1:BASE_PARITY:d638705d48916d7b"} 08:35:14 [08:35:14.486] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d638705d48916d7b {"provingJobId":"1:BASE_PARITY:d638705d48916d7b"} 08:35:14 [08:35:14.487] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d638705d48916d7b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d638705d48916d7b"} 08:35:14 [08:35:14.487] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d638705d48916d7b type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:d638705d48916d7b"} 08:35:14 [08:35:14.489] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.489] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ee16f32d1a8467b2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ee16f32d1a8467b2"} 08:35:14 [08:35:14.490] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ee16f32d1a8467b2 {"provingJobId":"1:BASE_PARITY:ee16f32d1a8467b2"} 08:35:14 [08:35:14.490] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ee16f32d1a8467b2 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:ee16f32d1a8467b2"} 08:35:14 [08:35:14.492] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.493] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:22d5cd61c2c1104c epochNumber=1 {"provingJobId":"1:BASE_PARITY:22d5cd61c2c1104c"} 08:35:14 [08:35:14.494] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:22d5cd61c2c1104c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:22d5cd61c2c1104c"} 08:35:14 [08:35:14.494] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:22d5cd61c2c1104c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:22d5cd61c2c1104c"} 08:35:14 [08:35:14.496] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.497] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cb5c0e49bc60393f epochNumber=1 {"provingJobId":"1:BASE_PARITY:cb5c0e49bc60393f"} 08:35:14 [08:35:14.497] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cb5c0e49bc60393f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cb5c0e49bc60393f"} 08:35:14 [08:35:14.498] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cb5c0e49bc60393f type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:cb5c0e49bc60393f"} 08:35:14 [08:35:14.499] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.499] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8086fc948b28bad2 epochNumber=0 {"provingJobId":"1:BASE_PARITY:8086fc948b28bad2"} 08:35:14 [08:35:14.499] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8086fc948b28bad2 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8086fc948b28bad2"} 08:35:14 [08:35:14.500] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7696ff902e5c0037 epochNumber=0 {"provingJobId":"1:BASE_PARITY:7696ff902e5c0037"} 08:35:14 [08:35:14.500] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8086fc948b28bad2 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8086fc948b28bad2"} 08:35:14 [08:35:14.500] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5db3323670239012 epochNumber=0 {"provingJobId":"1:BASE_PARITY:5db3323670239012"} 08:35:14 [08:35:14.502] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.503] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.503] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8c4b907a87e02d14 epochNumber=2 {"provingJobId":"1:BASE_PARITY:8c4b907a87e02d14"} 08:35:14 [08:35:14.504] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d2e249c58391d1d1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d2e249c58391d1d1"} 08:35:14 [08:35:14.505] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.506] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6fed9a22d0399e80 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6fed9a22d0399e80"} 08:35:14 [08:35:14.507] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.508] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f43ad0c188fa2019 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f43ad0c188fa2019"} 08:35:14 [08:35:14.509] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c4d81f94939f2d4f epochNumber=1 {"provingJobId":"1:BASE_PARITY:c4d81f94939f2d4f"} 08:35:14 [08:35:14.511] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d3668f711862f8b2 epochNumber=2 {"provingJobId":"1:BASE_PARITY:d3668f711862f8b2"} 08:35:14 [08:35:14.511] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:96ae96164b2f2247 epochNumber=1 {"provingJobId":"1:BASE_PARITY:96ae96164b2f2247"} 08:35:14 [08:35:14.515] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.518] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2f9c85dd788cf473 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2f9c85dd788cf473"} 08:35:14 [08:35:14.523] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f4b251cb9b7517ae epochNumber=1 {"provingJobId":"1:BASE_PARITY:f4b251cb9b7517ae"} 08:35:14 [08:35:14.524] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a97f8a13a99afb8a epochNumber=2 {"provingJobId":"1:BASE_PARITY:a97f8a13a99afb8a"} 08:35:14 [08:35:14.525] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dafe831d60ad99ef epochNumber=1 {"provingJobId":"1:BASE_PARITY:dafe831d60ad99ef"} 08:35:14 [08:35:14.529] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.530] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea01599149a5c1ae epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea01599149a5c1ae"} 08:35:14 [08:35:14.531] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:337fa1e95a31595d epochNumber=1 {"provingJobId":"1:BASE_PARITY:337fa1e95a31595d"} 08:35:14 [08:35:14.532] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7790370ebeec2542 epochNumber=2 {"provingJobId":"1:BASE_PARITY:7790370ebeec2542"} 08:35:14 [08:35:14.532] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:df77f3acf1a6b337 epochNumber=1 {"provingJobId":"1:BASE_PARITY:df77f3acf1a6b337"} 08:35:14 [08:35:14.536] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.537] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dc63ee4c0d0904f5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dc63ee4c0d0904f5"} 08:35:14 [08:35:14.538] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:771eb415d4ab1a48 epochNumber=1 {"provingJobId":"1:BASE_PARITY:771eb415d4ab1a48"} 08:35:14 [08:35:14.538] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dc63ee4c0d0904f5 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:dc63ee4c0d0904f5"} 08:35:14 [08:35:14.540] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.541] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a928a54e68193ec7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a928a54e68193ec7"} 08:35:14 [08:35:14.541] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:89f9b25b0d2a777b epochNumber=1 {"provingJobId":"1:BASE_PARITY:89f9b25b0d2a777b"} 08:35:14 [08:35:14.541] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a928a54e68193ec7 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:a928a54e68193ec7"} 08:35:14 [08:35:14.543] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.544] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:87d4a3c0937a0300 epochNumber=1 {"provingJobId":"1:BASE_PARITY:87d4a3c0937a0300"} 08:35:14 [08:35:14.553] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:01772ce972930ec4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:01772ce972930ec4"} 08:35:14 [08:35:14.554] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:87d4a3c0937a0300 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:87d4a3c0937a0300"} 08:35:14 [08:35:14.555] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.556] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:41f2f267232e26d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:41f2f267232e26d6"} 08:35:14 [08:35:14.556] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:41f2f267232e26d6 {"provingJobId":"1:BASE_PARITY:41f2f267232e26d6"} 08:35:14 [08:35:14.557] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:41f2f267232e26d6 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:41f2f267232e26d6"} 08:35:14 [08:35:14.557] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0b35b193a3240fe9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0b35b193a3240fe9"} 08:35:14 [08:35:14.557] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:41f2f267232e26d6 has already been completed {"provingJobId":"1:BASE_PARITY:41f2f267232e26d6"} 08:35:14 [08:35:14.558] INFO: prover-client:proving-broker Proving Broker started 08:35:14 [08:35:14.559] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:daf8c4d455a80297 epochNumber=1 {"provingJobId":"1:BASE_PARITY:daf8c4d455a80297"} 08:35:14 [08:35:14.559] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:385141a37e33fa1e epochNumber=2 {"provingJobId":"1:BASE_PARITY:385141a37e33fa1e"} 08:35:15 [08:35:15.613] INFO: prover-client:proving-broker Proving Broker started 08:35:15 [08:35:15.614] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:daf8c4d455a80297 settled=false {"provingJobId":"1:BASE_PARITY:daf8c4d455a80297","status":"pending"} 08:35:15 [08:35:15.615] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:385141a37e33fa1e settled=false {"provingJobId":"1:BASE_PARITY:385141a37e33fa1e","status":"pending"} 08:35:15 [08:35:15.621] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:daf8c4d455a80297 type=BASE_PARITY startedAt=1749803714561 older agent has taken job {"provingJobId":"1:BASE_PARITY:daf8c4d455a80297"} 08:35:15 [08:35:15.622] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:daf8c4d455a80297 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:daf8c4d455a80297"} 08:35:15 [08:35:15.624] INFO: prover-client:proving-broker Proving Broker started 08:35:15 [08:35:15.628] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d178497973f4bcdc epochNumber=1 {"provingJobId":"1:BASE_PARITY:d178497973f4bcdc"} 08:35:15 [08:35:15.629] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:257fe8f070c9bd5b epochNumber=2 {"provingJobId":"1:BASE_PARITY:257fe8f070c9bd5b"} 08:35:16 [08:35:16.630] INFO: prover-client:proving-broker Proving Broker started 08:35:16 [08:35:16.631] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d178497973f4bcdc settled=false {"provingJobId":"1:BASE_PARITY:d178497973f4bcdc","status":"pending"} 08:35:16 [08:35:16.631] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:257fe8f070c9bd5b settled=false {"provingJobId":"1:BASE_PARITY:257fe8f070c9bd5b","status":"pending"} 08:35:16 [08:35:16.632] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d178497973f4bcdc type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:d178497973f4bcdc"} 08:35:16 [08:35:16.638] INFO: prover-client:proving-broker Proving Broker started 08:35:16 [08:35:16.639] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:18ec125cec99dec5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:18ec125cec99dec5"} 08:35:16 [08:35:16.639] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:74c9323fef4254f0 epochNumber=2 {"provingJobId":"1:BASE_PARITY:74c9323fef4254f0"} 08:35:17 [08:35:17.641] INFO: prover-client:proving-broker Proving Broker started 08:35:17 [08:35:17.642] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:18ec125cec99dec5 settled=false {"provingJobId":"1:BASE_PARITY:18ec125cec99dec5","status":"pending"} 08:35:17 [08:35:17.643] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:74c9323fef4254f0 settled=false {"provingJobId":"1:BASE_PARITY:74c9323fef4254f0","status":"pending"} 08:35:17 [08:35:17.644] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:18ec125cec99dec5 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:18ec125cec99dec5"} 08:35:17 [08:35:17.646] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:18ec125cec99dec5 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:18ec125cec99dec5"} 08:35:17 [08:35:17.648] INFO: prover-client:proving-broker Proving Broker started 08:35:17 [08:35:17.649] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:021db80eee3bc75f epochNumber=1 {"provingJobId":"1:BASE_PARITY:021db80eee3bc75f"} 08:35:17 [08:35:17.650] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cde740f79e6a715f epochNumber=2 {"provingJobId":"1:BASE_PARITY:cde740f79e6a715f"} 08:35:17 [08:35:17.651] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:021db80eee3bc75f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:021db80eee3bc75f"} 08:35:17 [08:35:17.652] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cde740f79e6a715f type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:cde740f79e6a715f"} 08:35:17 [08:35:17.654] INFO: prover-client:proving-broker Proving Broker started 08:35:17 [08:35:17.655] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:db8d4d77e8a54175 epochNumber=1 {"provingJobId":"1:BASE_PARITY:db8d4d77e8a54175"} 08:35:17 [08:35:17.656] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6c8ffab402a3b4e4 epochNumber=2 {"provingJobId":"1:BASE_PARITY:6c8ffab402a3b4e4"} 08:35:17 [08:35:17.656] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:db8d4d77e8a54175 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:db8d4d77e8a54175"} 08:35:17 [08:35:17.657] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:db8d4d77e8a54175 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:db8d4d77e8a54175"} 08:35:17 [08:35:17.658] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6c8ffab402a3b4e4 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:6c8ffab402a3b4e4"} 08:35:17 [08:35:17.659] INFO: prover-client:proving-broker Proving Broker started 08:35:17 [08:35:17.660] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:9f654c21feb72a65 err=test error {"provingJoId":"1:BASE_PARITY:9f654c21feb72a65"} 08:35:17 [08:35:17.662] INFO: prover-client:proving-broker Proving Broker started 08:35:17 [08:35:17.663] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d6b831dcc7b12337 not found {"provingJobId":"1:BASE_PARITY:d6b831dcc7b12337"} 08:35:17 [08:35:17.665] INFO: prover-client:proving-broker Proving Broker started 08:35:17 [08:35:17.666] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:29335efe452e0ebc epochNumber=1 {"provingJobId":"1:BASE_PARITY:29335efe452e0ebc"} 08:35:17 [08:35:17.667] INFO: prover-client:proving-broker Proving Broker started 08:35:17 [08:35:17.667] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1926075be650cd22 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1926075be650cd22"} 08:35:17 [08:35:17.769] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1926075be650cd22 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:1926075be650cd22"} 08:35:17 [08:35:17.776] INFO: prover-client:proving-broker Proving Broker started 08:35:17 [08:35:17.777] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c615da694722728b epochNumber=1 {"provingJobId":"1:BASE_PARITY:c615da694722728b"} 08:35:17 [08:35:17.879] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c615da694722728b timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:c615da694722728b"} 08:35:17 [08:35:17.884] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c4193db6f0ad95b0 epochNumber=10 {"provingJobId":"1:BASE_PARITY:c4193db6f0ad95b0"} 08:35:17 [08:35:17.905] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:18 [08:35:18.011] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.012] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:13f385e2adbc6070 epochNumber=1 {"provingJobId":"1:BASE_PARITY:13f385e2adbc6070"} 08:35:18 [08:35:18.165] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:13f385e2adbc6070 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:13f385e2adbc6070"} 08:35:18 [08:35:18.215] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.217] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:82ffd42622cf6930 epochNumber=1 {"provingJobId":"1:BASE_PARITY:82ffd42622cf6930"} 08:35:18 [08:35:18.218] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:82ffd42622cf6930 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:82ffd42622cf6930"} 08:35:18 [08:35:18.220] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.221] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aeaa228d91925596 epochNumber=1 {"provingJobId":"1:BASE_PARITY:aeaa228d91925596"} 08:35:18 [08:35:18.222] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:aeaa228d91925596 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:aeaa228d91925596"} 08:35:18 [08:35:18.222] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:aeaa228d91925596 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:aeaa228d91925596"} 08:35:18 [08:35:18.223] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.224] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab037477abc3ac29 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ab037477abc3ac29"} 08:35:18 [08:35:18.225] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ab037477abc3ac29 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ab037477abc3ac29"} 08:35:18 [08:35:18.226] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.226] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ccddb386a7985891 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ccddb386a7985891"} 08:35:18 [08:35:18.227] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ccddb386a7985891 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:ccddb386a7985891"} 08:35:18 [08:35:18.228] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9f6dd3607f2a7f65 epochNumber=3 {"provingJobId":"1:BASE_PARITY:9f6dd3607f2a7f65"} 08:35:18 [08:35:18.252] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:18 [08:35:18.254] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ccddb386a7985891 err=test error {"provingJoId":"1:BASE_PARITY:ccddb386a7985891"} 08:35:18 [08:35:18.256] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.256] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:82073de68e7599ee settled=false {"provingJobId":"1:BASE_PARITY:82073de68e7599ee","status":"pending"} 08:35:18 [08:35:18.256] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:80a1758c1594583f settled=false {"provingJobId":"1:BASE_PARITY:80a1758c1594583f","status":"pending"} 08:35:18 [08:35:18.258] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.258] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:31f844830263150d settled=true {"provingJobId":"1:BASE_PARITY:31f844830263150d","status":"fulfilled"} 08:35:18 [08:35:18.259] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:0a51ca32b0d0eab9 settled=true {"provingJobId":"2:BASE_PARITY:0a51ca32b0d0eab9","status":"fulfilled"} 08:35:18 [08:35:18.260] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.260] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f52be102cfeb44b2 settled=true {"provingJobId":"1:BASE_PARITY:f52be102cfeb44b2","status":"fulfilled"} 08:35:18 [08:35:18.260] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2a49243b16691c85 settled=false {"provingJobId":"1:BASE_PARITY:2a49243b16691c85","status":"pending"} 08:35:18 [08:35:18.261] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.262] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8616a38b3e29cbe8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8616a38b3e29cbe8"} 08:35:18 [08:35:18.264] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.264] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f56329c8f5f33d2c epochNumber=1 {"provingJobId":"1:BASE_PARITY:f56329c8f5f33d2c"} 08:35:18 [08:35:18.265] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:f56329c8f5f33d2c: Error: db error: Error: db error 08:35:18 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:f56329c8f5f33d2c"} 08:35:18 [08:35:18.266] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.267] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:73dc888c8b9d90e2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:73dc888c8b9d90e2"} 08:35:18 [08:35:18.267] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:73dc888c8b9d90e2 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:73dc888c8b9d90e2"} 08:35:18 [08:35:18.267] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:73dc888c8b9d90e2 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:73dc888c8b9d90e2"} 08:35:18 [08:35:18.268] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.268] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f924f445521ca2b1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f924f445521ca2b1"} 08:35:18 [08:35:18.269] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f924f445521ca2b1 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f924f445521ca2b1"} 08:35:18 [08:35:18.269] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f924f445521ca2b1 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f924f445521ca2b1"} 08:35:18 [08:35:18.269] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:f924f445521ca2b1: Error: db error 08:35:18 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:f924f445521ca2b1"} 08:35:18 [08:35:18.270] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.270] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d1b43e7d951bdf47 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d1b43e7d951bdf47"} 08:35:18 [08:35:18.271] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d1b43e7d951bdf47 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d1b43e7d951bdf47"} 08:35:18 [08:35:18.277] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d1b43e7d951bdf47 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:d1b43e7d951bdf47"} 08:35:18 [08:35:18.281] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.281] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a8919d0c943e2ebf epochNumber=1 {"provingJobId":"1:BASE_PARITY:a8919d0c943e2ebf"} 08:35:18 [08:35:18.282] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a8919d0c943e2ebf type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a8919d0c943e2ebf"} 08:35:18 [08:35:18.282] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a8919d0c943e2ebf type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:a8919d0c943e2ebf"} 08:35:18 [08:35:18.283] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a8919d0c943e2ebf jobErr=test error: Error: db error 08:35:18 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:a8919d0c943e2ebf"} 08:35:18 [08:35:18.284] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.284] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2ca6ea9f6df92203 not found {"provingJobId":"1:BASE_PARITY:2ca6ea9f6df92203"} 08:35:18 [08:35:18.285] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.286] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3217c7a505b919f2 err=test error {"provingJoId":"1:BASE_PARITY:3217c7a505b919f2"} 08:35:18 [08:35:18.290] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.302] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f0571690633c8154 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f0571690633c8154"} 08:35:18 [08:35:18.303] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f0571690633c8154 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f0571690633c8154"} 08:35:18 [08:35:18.303] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f0571690633c8154 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f0571690633c8154"} 08:35:18 [08:35:18.314] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:9808de7283b74e04 epochNumber=2 {"provingJobId":"2:BASE_PARITY:9808de7283b74e04"} 08:35:18 [08:35:18.315] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:9808de7283b74e04 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:9808de7283b74e04"} 08:35:18 [08:35:18.315] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:9808de7283b74e04 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:9808de7283b74e04"} 08:35:18 [08:35:18.377] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:8293960f28005c53 epochNumber=3 {"provingJobId":"3:BASE_PARITY:8293960f28005c53"} 08:35:18 [08:35:18.393] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:18 [08:35:18.438] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:339969f82e2021eb epochNumber=4 {"provingJobId":"4:BASE_PARITY:339969f82e2021eb"} 08:35:18 [08:35:18.445] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:18 [08:35:18.501] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:40595e82f2fd7606 epochNumber=5 {"provingJobId":"5:BASE_PARITY:40595e82f2fd7606"} 08:35:18 [08:35:18.522] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:18 [08:35:18.557] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.558] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6d0b1d307a5ed3d1 epochNumber=42 {"provingJobId":"1:BASE_PARITY:6d0b1d307a5ed3d1"} 08:35:18 [08:35:18.559] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testxeN3Oo/42 with map size 1073741824 08:35:18 [08:35:18.563] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.590] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:c12c2804e3d8fe5e epochNumber=1 {"provingJobId":"1:BASE_PARITY:c12c2804e3d8fe5e"} 08:35:18 [08:35:18.594] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.595] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:87f04abbb9a70a51 epochNumber=1 {"provingJobId":"1:BASE_PARITY:87f04abbb9a70a51"} 08:35:18 [08:35:18.595] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testneQmgy/1 with map size 1073741824 08:35:18 [08:35:18.601] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.629] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:93a920e1727a4bbf epochNumber=1 {"provingJobId":"1:BASE_PARITY:93a920e1727a4bbf"} 08:35:18 [08:35:18.641] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.642] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:42388efeaf8219fa epochNumber=1 {"provingJobId":"1:BASE_PARITY:42388efeaf8219fa"} 08:35:18 [08:35:18.642] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testw49BfM/1 with map size 1073741824 08:35:18 [08:35:18.644] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.661] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:42388efeaf8219fa epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:42388efeaf8219fa"} 08:35:18 [08:35:18.665] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.666] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5a6e6e87500b70d4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5a6e6e87500b70d4"} 08:35:18 [08:35:18.669] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpJzVnw/1 with map size 1073741824 08:35:18 [08:35:18.673] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.694] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5a6e6e87500b70d4 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:5a6e6e87500b70d4"} 08:35:18 [08:35:18.695] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5a6e6e87500b70d4 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5a6e6e87500b70d4"} 08:35:18 [08:35:18.701] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5a6e6e87500b70d4 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:5a6e6e87500b70d4"} 08:35:18 [08:35:18.704] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.704] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f6a9bffcf6468d28 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f6a9bffcf6468d28"} 08:35:18 [08:35:18.705] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8a53FA/1 with map size 1073741824 08:35:18 [08:35:18.707] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.733] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.736] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.737] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:50f187d7115a261a epochNumber=1 {"provingJobId":"1:BASE_PARITY:50f187d7115a261a"} 08:35:18 [08:35:18.738] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIQpknf/1 with map size 1073741824 08:35:18 [08:35:18.739] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.758] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:50f187d7115a261a {"provingJobId":"1:BASE_PARITY:50f187d7115a261a"} 08:35:18 [08:35:18.758] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:50f187d7115a261a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:50f187d7115a261a"} 08:35:18 [08:35:18.758] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:50f187d7115a261a type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:50f187d7115a261a"} 08:35:18 [08:35:18.767] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.768] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8923ea23be9412e3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8923ea23be9412e3"} 08:35:18 [08:35:18.768] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testliW2gc/1 with map size 1073741824 08:35:18 [08:35:18.770] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.787] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:8923ea23be9412e3 {"provingJobId":"1:BASE_PARITY:8923ea23be9412e3"} 08:35:18 [08:35:18.788] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8923ea23be9412e3 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:8923ea23be9412e3"} 08:35:18 [08:35:18.797] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.798] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0deb53b0a86d0f4f epochNumber=1 {"provingJobId":"1:BASE_PARITY:0deb53b0a86d0f4f"} 08:35:18 [08:35:18.798] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4geEe4/1 with map size 1073741824 08:35:18 [08:35:18.800] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.823] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0deb53b0a86d0f4f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0deb53b0a86d0f4f"} 08:35:18 [08:35:18.824] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0deb53b0a86d0f4f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0deb53b0a86d0f4f"} 08:35:18 [08:35:18.840] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.841] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9eca817f385d3132 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9eca817f385d3132"} 08:35:18 [08:35:18.842] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHIsdxv/1 with map size 1073741824 08:35:18 [08:35:18.844] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.861] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9eca817f385d3132 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9eca817f385d3132"} 08:35:18 [08:35:18.862] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9eca817f385d3132 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9eca817f385d3132"} 08:35:18 [08:35:18.870] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.870] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f36b7341b31ccf21 epochNumber=0 {"provingJobId":"1:BASE_PARITY:f36b7341b31ccf21"} 08:35:18 [08:35:18.871] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testzS9ZrL/0 with map size 1073741824 08:35:18 [08:35:18.872] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.897] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f36b7341b31ccf21 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f36b7341b31ccf21"} 08:35:18 [08:35:18.897] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cf7b50ce60103280 epochNumber=0 {"provingJobId":"1:BASE_PARITY:cf7b50ce60103280"} 08:35:18 [08:35:18.898] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f36b7341b31ccf21 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f36b7341b31ccf21"} 08:35:18 [08:35:18.898] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:01144c7e72323299 epochNumber=0 {"provingJobId":"1:BASE_PARITY:01144c7e72323299"} 08:35:18 [08:35:18.911] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.914] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.915] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:869efe92da4cd5d5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:869efe92da4cd5d5"} 08:35:18 [08:35:18.916] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrtSbRl/2 with map size 1073741824 08:35:18 [08:35:18.922] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.940] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bd5cafb68404bf7d epochNumber=1 {"provingJobId":"1:BASE_PARITY:bd5cafb68404bf7d"} 08:35:18 [08:35:18.941] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrtSbRl/1 with map size 1073741824 08:35:18 [08:35:18.943] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.963] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.964] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d8a388904e807b3e epochNumber=1 {"provingJobId":"1:BASE_PARITY:d8a388904e807b3e"} 08:35:18 [08:35:18.964] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvd2g62/1 with map size 1073741824 08:35:18 [08:35:18.970] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:18 [08:35:18.992] INFO: prover-client:proving-broker Proving Broker started 08:35:18 [08:35:18.992] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ad6982ec3123f1d epochNumber=1 {"provingJobId":"1:BASE_PARITY:8ad6982ec3123f1d"} 08:35:18 [08:35:18.994] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKhDAXk/1 with map size 1073741824 08:35:18 [08:35:18.996] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.012] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ef9e057f6244efc epochNumber=1 {"provingJobId":"1:BASE_PARITY:8ef9e057f6244efc"} 08:35:19 [08:35:19.023] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0ebc4bbafac9747f epochNumber=2 {"provingJobId":"1:BASE_PARITY:0ebc4bbafac9747f"} 08:35:19 [08:35:19.024] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKhDAXk/2 with map size 1073741824 08:35:19 [08:35:19.029] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.051] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:69b1c2d82feb8a81 epochNumber=1 {"provingJobId":"1:BASE_PARITY:69b1c2d82feb8a81"} 08:35:19 [08:35:19.062] INFO: prover-client:proving-broker Proving Broker started 08:35:19 [08:35:19.063] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5fefe9f9d36a48ef epochNumber=1 {"provingJobId":"1:BASE_PARITY:5fefe9f9d36a48ef"} 08:35:19 [08:35:19.064] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test81PLkc/1 with map size 1073741824 08:35:19 [08:35:19.066] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.091] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e472f46e7048ecde epochNumber=1 {"provingJobId":"1:BASE_PARITY:e472f46e7048ecde"} 08:35:19 [08:35:19.100] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:810e34d3a1ae39fc epochNumber=2 {"provingJobId":"1:BASE_PARITY:810e34d3a1ae39fc"} 08:35:19 [08:35:19.101] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test81PLkc/2 with map size 1073741824 08:35:19 [08:35:19.103] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.127] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e6a85fb9e9113181 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e6a85fb9e9113181"} 08:35:19 [08:35:19.138] INFO: prover-client:proving-broker Proving Broker started 08:35:19 [08:35:19.139] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cb4682fa41f076e0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cb4682fa41f076e0"} 08:35:19 [08:35:19.139] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJ0YQKH/1 with map size 1073741824 08:35:19 [08:35:19.142] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.160] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a44dd47c48fa302b epochNumber=1 {"provingJobId":"1:BASE_PARITY:a44dd47c48fa302b"} 08:35:19 [08:35:19.165] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3b70d3756d6fb209 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3b70d3756d6fb209"} 08:35:19 [08:35:19.166] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJ0YQKH/2 with map size 1073741824 08:35:19 [08:35:19.167] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.186] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:29b906802dd9fcd0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:29b906802dd9fcd0"} 08:35:19 [08:35:19.196] INFO: prover-client:proving-broker Proving Broker started 08:35:19 [08:35:19.196] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0bc451fb7a03c52b epochNumber=1 {"provingJobId":"1:BASE_PARITY:0bc451fb7a03c52b"} 08:35:19 [08:35:19.198] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3iEH31/1 with map size 1073741824 08:35:19 [08:35:19.200] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.221] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f6ab23875b9b9645 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f6ab23875b9b9645"} 08:35:19 [08:35:19.229] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0bc451fb7a03c52b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0bc451fb7a03c52b"} 08:35:19 [08:35:19.242] INFO: prover-client:proving-broker Proving Broker started 08:35:19 [08:35:19.243] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1cfc95e05c90c245 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1cfc95e05c90c245"} 08:35:19 [08:35:19.244] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAoUzhF/1 with map size 1073741824 08:35:19 [08:35:19.247] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.264] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:515a948cb689efb0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:515a948cb689efb0"} 08:35:19 [08:35:19.270] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1cfc95e05c90c245 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:1cfc95e05c90c245"} 08:35:19 [08:35:19.281] INFO: prover-client:proving-broker Proving Broker started 08:35:19 [08:35:19.282] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:38f1fa754eb0dd70 epochNumber=1 {"provingJobId":"1:BASE_PARITY:38f1fa754eb0dd70"} 08:35:19 [08:35:19.282] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrQMKVL/1 with map size 1073741824 08:35:19 [08:35:19.286] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.304] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7d95020a24e3a561 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7d95020a24e3a561"} 08:35:19 [08:35:19.309] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:38f1fa754eb0dd70 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:38f1fa754eb0dd70"} 08:35:19 [08:35:19.313] INFO: prover-client:proving-broker Proving Broker started 08:35:19 [08:35:19.314] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:07be1b7558f0dfc5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:07be1b7558f0dfc5"} 08:35:19 [08:35:19.314] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwjwCJT/1 with map size 1073741824 08:35:19 [08:35:19.320] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.340] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:07be1b7558f0dfc5 {"provingJobId":"1:BASE_PARITY:07be1b7558f0dfc5"} 08:35:19 [08:35:19.340] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:07be1b7558f0dfc5 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:07be1b7558f0dfc5"} 08:35:19 [08:35:19.346] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b5ed000980dc614a epochNumber=1 {"provingJobId":"1:BASE_PARITY:b5ed000980dc614a"} 08:35:19 [08:35:19.352] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:07be1b7558f0dfc5 has already been completed {"provingJobId":"1:BASE_PARITY:07be1b7558f0dfc5"} 08:35:19 [08:35:19.356] INFO: prover-client:proving-broker Proving Broker started 08:35:19 [08:35:19.357] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2d256328ee28cd08 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2d256328ee28cd08"} 08:35:19 [08:35:19.357] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0zELIM/1 with map size 1073741824 08:35:19 [08:35:19.358] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:19 [08:35:19.373] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:04c01fcb3c470f03 epochNumber=2 {"provingJobId":"1:BASE_PARITY:04c01fcb3c470f03"} 08:35:19 [08:35:19.374] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0zELIM/2 with map size 1073741824 08:35:19 [08:35:19.376] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:20 [08:35:20.442] INFO: prover-client:proving-broker Proving Broker started 08:35:20 [08:35:20.448] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2d256328ee28cd08 settled=false {"provingJobId":"1:BASE_PARITY:2d256328ee28cd08","status":"pending"} 08:35:20 [08:35:20.450] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:04c01fcb3c470f03 settled=false {"provingJobId":"1:BASE_PARITY:04c01fcb3c470f03","status":"pending"} 08:35:20 [08:35:20.451] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:2d256328ee28cd08 type=BASE_PARITY startedAt=1749803719390 older agent has taken job {"provingJobId":"1:BASE_PARITY:2d256328ee28cd08"} 08:35:20 [08:35:20.451] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2d256328ee28cd08 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:2d256328ee28cd08"} 08:35:20 [08:35:20.455] INFO: prover-client:proving-broker Proving Broker started 08:35:20 [08:35:20.456] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:385e451cc97390bd epochNumber=1 {"provingJobId":"1:BASE_PARITY:385e451cc97390bd"} 08:35:20 [08:35:20.456] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8M6nvG/1 with map size 1073741824 08:35:20 [08:35:20.458] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:20 [08:35:20.477] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3fad324ae99c3513 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3fad324ae99c3513"} 08:35:20 [08:35:20.478] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8M6nvG/2 with map size 1073741824 08:35:20 [08:35:20.479] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:21 [08:35:21.496] INFO: prover-client:proving-broker Proving Broker started 08:35:21 [08:35:21.498] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:385e451cc97390bd settled=false {"provingJobId":"1:BASE_PARITY:385e451cc97390bd","status":"pending"} 08:35:21 [08:35:21.500] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3fad324ae99c3513 settled=false {"provingJobId":"1:BASE_PARITY:3fad324ae99c3513","status":"pending"} 08:35:21 [08:35:21.501] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:385e451cc97390bd type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:385e451cc97390bd"} 08:35:21 [08:35:21.505] INFO: prover-client:proving-broker Proving Broker started 08:35:21 [08:35:21.506] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:66464e7c8d2aae94 epochNumber=1 {"provingJobId":"1:BASE_PARITY:66464e7c8d2aae94"} 08:35:21 [08:35:21.507] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeJ1xqT/1 with map size 1073741824 08:35:21 [08:35:21.509] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:21 [08:35:21.529] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03a01b3bc79550cf epochNumber=2 {"provingJobId":"1:BASE_PARITY:03a01b3bc79550cf"} 08:35:21 [08:35:21.530] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testeJ1xqT/2 with map size 1073741824 08:35:21 [08:35:21.532] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:22 [08:35:22.548] INFO: prover-client:proving-broker Proving Broker started 08:35:22 [08:35:22.553] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:66464e7c8d2aae94 settled=false {"provingJobId":"1:BASE_PARITY:66464e7c8d2aae94","status":"pending"} 08:35:22 [08:35:22.555] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:03a01b3bc79550cf settled=false {"provingJobId":"1:BASE_PARITY:03a01b3bc79550cf","status":"pending"} 08:35:22 [08:35:22.556] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:66464e7c8d2aae94 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:66464e7c8d2aae94"} 08:35:22 [08:35:22.556] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:66464e7c8d2aae94 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:66464e7c8d2aae94"} 08:35:22 [08:35:22.566] INFO: prover-client:proving-broker Proving Broker started 08:35:22 [08:35:22.567] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bbb34cf083160faa epochNumber=1 {"provingJobId":"1:BASE_PARITY:bbb34cf083160faa"} 08:35:22 [08:35:22.568] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAxxFmY/1 with map size 1073741824 08:35:22 [08:35:22.572] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:22 [08:35:22.598] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:78a336120558cad9 epochNumber=2 {"provingJobId":"1:BASE_PARITY:78a336120558cad9"} 08:35:22 [08:35:22.601] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAxxFmY/2 with map size 1073741824 08:35:22 [08:35:22.607] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:22 [08:35:22.638] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bbb34cf083160faa type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:bbb34cf083160faa"} 08:35:22 [08:35:22.646] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:78a336120558cad9 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:78a336120558cad9"} 08:35:22 [08:35:22.657] INFO: prover-client:proving-broker Proving Broker started 08:35:22 [08:35:22.658] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2eadd22e775e3ec2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2eadd22e775e3ec2"} 08:35:22 [08:35:22.660] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnQKHJr/1 with map size 1073741824 08:35:22 [08:35:22.666] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:22 [08:35:22.684] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:31c32e412fb7d54b epochNumber=2 {"provingJobId":"1:BASE_PARITY:31c32e412fb7d54b"} 08:35:22 [08:35:22.686] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnQKHJr/2 with map size 1073741824 08:35:22 [08:35:22.689] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:22 [08:35:22.714] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2eadd22e775e3ec2 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2eadd22e775e3ec2"} 08:35:22 [08:35:22.714] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2eadd22e775e3ec2 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2eadd22e775e3ec2"} 08:35:22 [08:35:22.724] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:31c32e412fb7d54b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:31c32e412fb7d54b"} 08:35:22 [08:35:22.739] INFO: prover-client:proving-broker Proving Broker started 08:35:22 [08:35:22.740] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:23bd6505609e4b9f err=test error {"provingJoId":"1:BASE_PARITY:23bd6505609e4b9f"} 08:35:22 [08:35:22.744] INFO: prover-client:proving-broker Proving Broker started 08:35:22 [08:35:22.745] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a56244ea4c440d11 not found {"provingJobId":"1:BASE_PARITY:a56244ea4c440d11"} 08:35:22 [08:35:22.747] INFO: prover-client:proving-broker Proving Broker started 08:35:22 [08:35:22.748] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:43c1f52e0eb473fc epochNumber=1 {"provingJobId":"1:BASE_PARITY:43c1f52e0eb473fc"} 08:35:22 [08:35:22.748] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlOKrXG/1 with map size 1073741824 08:35:22 [08:35:22.752] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:22 [08:35:22.773] INFO: prover-client:proving-broker Proving Broker started 08:35:22 [08:35:22.774] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8e72659c78490cf2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8e72659c78490cf2"} 08:35:22 [08:35:22.775] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKHRnqe/1 with map size 1073741824 08:35:22 [08:35:22.777] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:22 [08:35:22.906] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8e72659c78490cf2 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:8e72659c78490cf2"} 08:35:22 [08:35:22.917] INFO: prover-client:proving-broker Proving Broker started 08:35:22 [08:35:22.918] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:854e7505ff2feede epochNumber=1 {"provingJobId":"1:BASE_PARITY:854e7505ff2feede"} 08:35:22 [08:35:22.919] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4xZsP0/1 with map size 1073741824 08:35:22 [08:35:22.921] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.043] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:854e7505ff2feede timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:854e7505ff2feede"} 08:35:23 [08:35:23.049] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:34462927639d60be epochNumber=10 {"provingJobId":"1:BASE_PARITY:34462927639d60be"} 08:35:23 [08:35:23.050] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test4xZsP0/10 with map size 1073741824 08:35:23 [08:35:23.052] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.069] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:23 [08:35:23.070] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 08:35:23 [08:35:23.196] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.197] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d5027eaa3657b424 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d5027eaa3657b424"} 08:35:23 [08:35:23.198] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testj8Zz39/1 with map size 1073741824 08:35:23 [08:35:23.199] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.372] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d5027eaa3657b424 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:d5027eaa3657b424"} 08:35:23 [08:35:23.433] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.434] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:679dc07b864e22fe epochNumber=1 {"provingJobId":"1:BASE_PARITY:679dc07b864e22fe"} 08:35:23 [08:35:23.434] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpODmHc/1 with map size 1073741824 08:35:23 [08:35:23.438] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.459] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:679dc07b864e22fe type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:679dc07b864e22fe"} 08:35:23 [08:35:23.463] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.463] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6af2b8ccdf9332bd epochNumber=1 {"provingJobId":"1:BASE_PARITY:6af2b8ccdf9332bd"} 08:35:23 [08:35:23.464] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrFKeVq/1 with map size 1073741824 08:35:23 [08:35:23.466] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.483] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6af2b8ccdf9332bd type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:6af2b8ccdf9332bd"} 08:35:23 [08:35:23.484] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6af2b8ccdf9332bd type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:6af2b8ccdf9332bd"} 08:35:23 [08:35:23.493] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.494] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7b849ad63f49864a epochNumber=1 {"provingJobId":"1:BASE_PARITY:7b849ad63f49864a"} 08:35:23 [08:35:23.495] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testChNer5/1 with map size 1073741824 08:35:23 [08:35:23.499] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.516] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7b849ad63f49864a type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:7b849ad63f49864a"} 08:35:23 [08:35:23.531] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.532] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a65ce8d7f2398912 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a65ce8d7f2398912"} 08:35:23 [08:35:23.532] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCunl2a/1 with map size 1073741824 08:35:23 [08:35:23.535] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.551] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a65ce8d7f2398912 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:a65ce8d7f2398912"} 08:35:23 [08:35:23.553] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:257eec0697301fdd epochNumber=3 {"provingJobId":"1:BASE_PARITY:257eec0697301fdd"} 08:35:23 [08:35:23.554] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testCunl2a/3 with map size 1073741824 08:35:23 [08:35:23.557] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.582] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:23 [08:35:23.582] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 08:35:23 [08:35:23.601] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a65ce8d7f2398912 err=test error {"provingJoId":"1:BASE_PARITY:a65ce8d7f2398912"} 08:35:23 [08:35:23.606] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCnBwjp/1 with map size 1073741824 08:35:23 [08:35:23.608] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.642] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCnBwjp/2 with map size 1073741824 08:35:23 [08:35:23.648] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.672] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.675] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:035d62a3144bdce4 settled=false {"provingJobId":"1:BASE_PARITY:035d62a3144bdce4","status":"pending"} 08:35:23 [08:35:23.677] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:45dfcba4b18073b4 settled=false {"provingJobId":"1:BASE_PARITY:45dfcba4b18073b4","status":"pending"} 08:35:23 [08:35:23.685] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqexRLN/1 with map size 1073741824 08:35:23 [08:35:23.687] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.706] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqexRLN/2 with map size 1073741824 08:35:23 [08:35:23.708] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.741] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.745] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f8515c5693083598 settled=true {"provingJobId":"1:BASE_PARITY:f8515c5693083598","status":"fulfilled"} 08:35:23 [08:35:23.748] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:cbe1d75f175466d9 settled=true {"provingJobId":"2:BASE_PARITY:cbe1d75f175466d9","status":"fulfilled"} 08:35:23 [08:35:23.759] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testc7cAA6/1 with map size 1073741824 08:35:23 [08:35:23.763] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.788] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testc7cAA6/2 with map size 1073741824 08:35:23 [08:35:23.791] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.816] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.818] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5707896a0cd1dba0 settled=true {"provingJobId":"1:BASE_PARITY:5707896a0cd1dba0","status":"fulfilled"} 08:35:23 [08:35:23.822] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:46135f97a897c098 settled=false {"provingJobId":"1:BASE_PARITY:46135f97a897c098","status":"pending"} 08:35:23 [08:35:23.830] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.832] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:930149ae11f094dd epochNumber=1 {"provingJobId":"1:BASE_PARITY:930149ae11f094dd"} 08:35:23 [08:35:23.835] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5BJ06u/1 with map size 1073741824 08:35:23 [08:35:23.840] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.865] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.866] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ccde1b9db0ebe4f3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ccde1b9db0ebe4f3"} 08:35:23 [08:35:23.867] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:ccde1b9db0ebe4f3: Error: db error: Error: db error 08:35:23 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:ccde1b9db0ebe4f3"} 08:35:23 [08:35:23.871] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.871] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:02b8babdd4fed52b epochNumber=1 {"provingJobId":"1:BASE_PARITY:02b8babdd4fed52b"} 08:35:23 [08:35:23.872] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testApxFKy/1 with map size 1073741824 08:35:23 [08:35:23.878] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.899] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:02b8babdd4fed52b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:02b8babdd4fed52b"} 08:35:23 [08:35:23.900] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:02b8babdd4fed52b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:02b8babdd4fed52b"} 08:35:23 [08:35:23.910] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.911] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ff88448da4d052f8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ff88448da4d052f8"} 08:35:23 [08:35:23.915] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvnkaRt/1 with map size 1073741824 08:35:23 [08:35:23.924] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.945] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ff88448da4d052f8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ff88448da4d052f8"} 08:35:23 [08:35:23.946] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ff88448da4d052f8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ff88448da4d052f8"} 08:35:23 [08:35:23.947] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:ff88448da4d052f8: Error: db error 08:35:23 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:ff88448da4d052f8"} 08:35:23 [08:35:23.952] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.953] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d8f3cdd885f0f1ce epochNumber=1 {"provingJobId":"1:BASE_PARITY:d8f3cdd885f0f1ce"} 08:35:23 [08:35:23.954] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYTK3Fa/1 with map size 1073741824 08:35:23 [08:35:23.957] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:23 [08:35:23.975] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d8f3cdd885f0f1ce type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d8f3cdd885f0f1ce"} 08:35:23 [08:35:23.976] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d8f3cdd885f0f1ce type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:d8f3cdd885f0f1ce"} 08:35:23 [08:35:23.984] INFO: prover-client:proving-broker Proving Broker started 08:35:23 [08:35:23.985] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7cbbe8eb45f564d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7cbbe8eb45f564d6"} 08:35:23 [08:35:23.985] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6GGf42/1 with map size 1073741824 08:35:23 [08:35:23.989] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:24 [08:35:24.011] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7cbbe8eb45f564d6 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:7cbbe8eb45f564d6"} 08:35:24 [08:35:24.011] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7cbbe8eb45f564d6 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:7cbbe8eb45f564d6"} 08:35:24 [08:35:24.012] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:7cbbe8eb45f564d6 jobErr=test error: Error: db error 08:35:24 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:7cbbe8eb45f564d6"} 08:35:24 [08:35:24.019] INFO: prover-client:proving-broker Proving Broker started 08:35:24 [08:35:24.020] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:eee334d185b175ec not found {"provingJobId":"1:BASE_PARITY:eee334d185b175ec"} 08:35:24 [08:35:24.028] INFO: prover-client:proving-broker Proving Broker started 08:35:24 [08:35:24.029] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:cfd42cf385d2d8e2 err=test error {"provingJoId":"1:BASE_PARITY:cfd42cf385d2d8e2"} 08:35:24 [08:35:24.033] INFO: prover-client:proving-broker Proving Broker started 08:35:24 [08:35:24.043] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5f4f2088e9aa18a8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5f4f2088e9aa18a8"} 08:35:24 [08:35:24.044] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnJppt7/1 with map size 1073741824 08:35:24 [08:35:24.051] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:24 [08:35:24.069] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5f4f2088e9aa18a8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5f4f2088e9aa18a8"} 08:35:24 [08:35:24.069] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5f4f2088e9aa18a8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5f4f2088e9aa18a8"} 08:35:24 [08:35:24.086] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:c72c5444918d9d5a epochNumber=2 {"provingJobId":"2:BASE_PARITY:c72c5444918d9d5a"} 08:35:24 [08:35:24.088] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnJppt7/2 with map size 1073741824 08:35:24 [08:35:24.089] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:24 [08:35:24.106] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:c72c5444918d9d5a type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:c72c5444918d9d5a"} 08:35:24 [08:35:24.107] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c72c5444918d9d5a type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:c72c5444918d9d5a"} 08:35:24 [08:35:24.172] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:b0cac3b55640867e epochNumber=3 {"provingJobId":"3:BASE_PARITY:b0cac3b55640867e"} 08:35:24 [08:35:24.173] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testnJppt7/3 with map size 1073741824 08:35:24 [08:35:24.175] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:24 [08:35:24.212] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:24 [08:35:24.212] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 08:35:24 [08:35:24.259] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:1f9311b94ef415e2 epochNumber=4 {"provingJobId":"4:BASE_PARITY:1f9311b94ef415e2"} 08:35:24 [08:35:24.261] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testnJppt7/4 with map size 1073741824 08:35:24 [08:35:24.264] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:24 [08:35:24.292] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:24 [08:35:24.292] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 08:35:24 [08:35:24.346] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:556eeca4f7097da3 epochNumber=5 {"provingJobId":"5:BASE_PARITY:556eeca4f7097da3"} 08:35:24 [08:35:24.347] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testnJppt7/5 with map size 1073741824 08:35:24 [08:35:24.349] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 08:35:24 [08:35:24.395] INFO: prover-client:proving-broker Cleaned up jobs=1 08:35:24 [08:35:24.396] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 08:35:24 PASS src/proving_broker/proving_broker.test.ts (13.195 s) 08:35:24 ProvingBroker 08:35:24 Producer API 08:35:24 ✓ refuses stale jobs (65 ms) 08:35:24 ✓ enqueues jobs (11 ms) 08:35:24 ✓ ignores duplicate jobs (6 ms) 08:35:24 ✓ reports correct status when enqueuing repeat jobs (7 ms) 08:35:24 ✓ throws an error in case of duplicate job IDs (33 ms) 08:35:24 ✓ returns not-found status for non-existing jobs (3 ms) 08:35:24 ✓ cancels jobs in queue (4 ms) 08:35:24 ✓ cancels jobs in-progress (3 ms) 08:35:24 ✓ returns job result if successful (4 ms) 08:35:24 ✓ returns job error if failed (2 ms) 08:35:24 ✓ correctly returns job status for concurrent writes (3 ms) 08:35:24 ✓ refuses stale jobs (38 ms) 08:35:24 ✓ enqueues jobs (46 ms) 08:35:24 ✓ ignores duplicate jobs (23 ms) 08:35:24 ✓ reports correct status when enqueuing repeat jobs (39 ms) 08:35:24 ✓ throws an error in case of duplicate job IDs (27 ms) 08:35:24 ✓ returns not-found status for non-existing jobs (4 ms) 08:35:24 ✓ cancels jobs in queue (30 ms) 08:35:24 ✓ cancels jobs in-progress (28 ms) 08:35:24 ✓ returns job result if successful (41 ms) 08:35:24 ✓ returns job error if failed (32 ms) 08:35:24 ✓ correctly returns job status for concurrent writes (41 ms) 08:35:24 Consumer API 08:35:24 ✓ returns undefined if no jobs are available (1 ms) 08:35:24 ✓ returns jobs in priority order (2 ms) 08:35:24 ✓ returns undefined if no jobs are available for the given allowList (2 ms) 08:35:24 ✓ returns a job if it is in the allowList (7 ms) 08:35:24 ✓ returns the most important job if it is in the allowList (13 ms) 08:35:24 ✓ returns any job if filter is empty (6 ms) 08:35:24 ✓ returns a new job when reporting job success (4 ms) 08:35:24 ✓ returns a new job when reporting permanent error (2 ms) 08:35:24 ✓ returns a new job when reporting retry-able error (13 ms) 08:35:24 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 08:35:24 ✓ returns a new job if job is already in progress elsewhere (1065 ms) 08:35:24 ✓ avoids sending the same job to a new agent after a restart (1010 ms) 08:35:24 ✓ avoids sending a completed job to a new agent after a restart (1010 ms) 08:35:24 ✓ tracks job result if in progress (6 ms) 08:35:24 ✓ tracks job result even if job is in queue (5 ms) 08:35:24 ✓ ignores reported job error if unknown job (2 ms) 08:35:24 ✓ ignores job result if unknown job (3 ms) 08:35:24 ✓ returns undefined if no jobs are available (2 ms) 08:35:24 ✓ returns jobs in priority order (48 ms) 08:35:24 ✓ returns undefined if no jobs are available for the given allowList (28 ms) 08:35:24 ✓ returns a job if it is in the allowList (70 ms) 08:35:24 ✓ returns the most important job if it is in the allowList (74 ms) 08:35:24 ✓ returns any job if filter is empty (59 ms) 08:35:24 ✓ returns a new job when reporting job success (44 ms) 08:35:24 ✓ returns a new job when reporting permanent error (39 ms) 08:35:24 ✓ returns a new job when reporting retry-able error (32 ms) 08:35:24 ✓ returns a new job when reporting progress if current one is cancelled (42 ms) 08:35:24 ✓ returns a new job if job is already in progress elsewhere (1098 ms) 08:35:24 ✓ avoids sending the same job to a new agent after a restart (1050 ms) 08:35:24 ✓ avoids sending a completed job to a new agent after a restart (1060 ms) 08:35:24 ✓ tracks job result if in progress (91 ms) 08:35:24 ✓ tracks job result even if job is in queue (79 ms) 08:35:24 ✓ ignores reported job error if unknown job (7 ms) 08:35:24 ✓ ignores job result if unknown job (4 ms) 08:35:24 Timeouts 08:35:24 ✓ tracks in progress jobs (1 ms) 08:35:24 ✓ re-enqueues jobs that time out (108 ms) 08:35:24 ✓ cancel stale jobs that time out (235 ms) 08:35:24 ✓ keeps the jobs in progress while it is alive (202 ms) 08:35:24 ✓ tracks in progress jobs (25 ms) 08:35:24 ✓ re-enqueues jobs that time out (140 ms) 08:35:24 ✓ cancel stale jobs that time out (283 ms) 08:35:24 ✓ keeps the jobs in progress while it is alive (233 ms) 08:35:24 Retries 08:35:24 ✓ retries jobs (5 ms) 08:35:24 ✓ retries up to a maximum number of times (4 ms) 08:35:24 ✓ passing retry=false does not retry (2 ms) 08:35:24 ✓ does not retry if job is stale (28 ms) 08:35:24 ✓ retries jobs (33 ms) 08:35:24 ✓ retries up to a maximum number of times (30 ms) 08:35:24 ✓ passing retry=false does not retry (37 ms) 08:35:24 ✓ does not retry if job is stale (74 ms) 08:35:24 Database management 08:35:24 ✓ re-enqueues proof requests on start (2 ms) 08:35:24 ✓ restores proof results on start (1 ms) 08:35:24 ✓ only re-enqueues unfinished jobs (2 ms) 08:35:24 ✓ saves job when enqueued (2 ms) 08:35:24 ✓ does not retain job if database fails to save (2 ms) 08:35:24 ✓ saves job result (2 ms) 08:35:24 ✓ saves result even if database fails to save (2 ms) 08:35:24 ✓ saves job error (9 ms) 08:35:24 ✓ saves job error even if database fails to save (3 ms) 08:35:24 ✓ does not save job result if job is unknown (2 ms) 08:35:24 ✓ does not save job error if job is unknown (1 ms) 08:35:24 ✓ cleans up old jobs periodically (264 ms) 08:35:24 ✓ re-enqueues proof requests on start (77 ms) 08:35:24 ✓ restores proof results on start (70 ms) 08:35:24 ✓ only re-enqueues unfinished jobs (74 ms) 08:35:24 ✓ saves job when enqueued (34 ms) 08:35:24 ✓ does not retain job if database fails to save (6 ms) 08:35:24 ✓ saves job result (40 ms) 08:35:24 ✓ saves result even if database fails to save (41 ms) 08:35:24 ✓ saves job error (33 ms) 08:35:24 ✓ saves job error even if database fails to save (32 ms) 08:35:24 ✓ does not save job result if job is unknown (8 ms) 08:35:24 ✓ does not save job error if job is unknown (8 ms) 08:35:24 ✓ cleans up old jobs periodically (398 ms) 08:35:24 08:35:24 Test Suites: 1 passed, 1 total 08:35:24 Tests: 96 passed, 96 total 08:35:24 Snapshots: 0 total 08:35:24 Time: 13.288 s 08:35:24 Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i. 08:35:24 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?