Parent Log: http://ci.aztec-labs.com/0592da4f2cb67ea9 Command: 411627cf9c043d00 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/721a41c449eb0c4538a26180f6061f326128cf78 Env: REF_NAME=gh-readonly-queue/next/pr-14963-60e60f99f2634ad75a76b6e103858ff9ccd041d5 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Mon Jun 16 18:41:43 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14963_amd64_x2-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 18:41:48 [18:41:48.737] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.744] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1e8f2e44b9ebd118 epochNumber=42 {"provingJobId":"1:BASE_PARITY:1e8f2e44b9ebd118"} 18:41:48 [18:41:48.746] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:823389a9640ad134 epochNumber=1 {"provingJobId":"1:BASE_PARITY:823389a9640ad134"} 18:41:48 [18:41:48.776] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.777] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:51704ce8f4fad7ac epochNumber=1 {"provingJobId":"1:BASE_PARITY:51704ce8f4fad7ac"} 18:41:48 [18:41:48.778] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:af96575ebd8299d4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:af96575ebd8299d4"} 18:41:48 [18:41:48.779] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.780] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a790f9bd359199c0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a790f9bd359199c0"} 18:41:48 [18:41:48.781] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a790f9bd359199c0 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:a790f9bd359199c0"} 18:41:48 [18:41:48.783] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.784] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2649d7d6883f22b3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2649d7d6883f22b3"} 18:41:48 [18:41:48.785] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2649d7d6883f22b3 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:2649d7d6883f22b3"} 18:41:48 [18:41:48.786] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2649d7d6883f22b3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2649d7d6883f22b3"} 18:41:48 [18:41:48.787] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2649d7d6883f22b3 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:2649d7d6883f22b3"} 18:41:48 [18:41:48.788] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.789] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:68f51252aaa64713 epochNumber=1 {"provingJobId":"1:BASE_PARITY:68f51252aaa64713"} 18:41:48 [18:41:48.816] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.818] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.818] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d95f2456913ea40a epochNumber=1 {"provingJobId":"1:BASE_PARITY:d95f2456913ea40a"} 18:41:48 [18:41:48.819] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d95f2456913ea40a {"provingJobId":"1:BASE_PARITY:d95f2456913ea40a"} 18:41:48 [18:41:48.819] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d95f2456913ea40a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d95f2456913ea40a"} 18:41:48 [18:41:48.820] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d95f2456913ea40a type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:d95f2456913ea40a"} 18:41:48 [18:41:48.821] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.821] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c97d012e38073c7f epochNumber=1 {"provingJobId":"1:BASE_PARITY:c97d012e38073c7f"} 18:41:48 [18:41:48.822] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c97d012e38073c7f {"provingJobId":"1:BASE_PARITY:c97d012e38073c7f"} 18:41:48 [18:41:48.822] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c97d012e38073c7f type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:c97d012e38073c7f"} 18:41:48 [18:41:48.823] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.824] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1388f414649c1294 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1388f414649c1294"} 18:41:48 [18:41:48.825] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1388f414649c1294 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:1388f414649c1294"} 18:41:48 [18:41:48.825] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1388f414649c1294 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:1388f414649c1294"} 18:41:48 [18:41:48.826] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.826] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c32628a717d9582b epochNumber=1 {"provingJobId":"1:BASE_PARITY:c32628a717d9582b"} 18:41:48 [18:41:48.827] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c32628a717d9582b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c32628a717d9582b"} 18:41:48 [18:41:48.827] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c32628a717d9582b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:c32628a717d9582b"} 18:41:48 [18:41:48.828] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.828] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f7c8de364ca57131 epochNumber=0 {"provingJobId":"1:BASE_PARITY:f7c8de364ca57131"} 18:41:48 [18:41:48.829] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f7c8de364ca57131 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f7c8de364ca57131"} 18:41:48 [18:41:48.829] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7726b844924e7b15 epochNumber=0 {"provingJobId":"1:BASE_PARITY:7726b844924e7b15"} 18:41:48 [18:41:48.829] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f7c8de364ca57131 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f7c8de364ca57131"} 18:41:48 [18:41:48.829] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0345662ab9b1cabc epochNumber=0 {"provingJobId":"1:BASE_PARITY:0345662ab9b1cabc"} 18:41:48 [18:41:48.830] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.831] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.832] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a8a3fe67d50b7b86 epochNumber=2 {"provingJobId":"1:BASE_PARITY:a8a3fe67d50b7b86"} 18:41:48 [18:41:48.832] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b566c872be7ab6ed epochNumber=1 {"provingJobId":"1:BASE_PARITY:b566c872be7ab6ed"} 18:41:48 [18:41:48.833] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.833] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c9151cb38d788871 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c9151cb38d788871"} 18:41:48 [18:41:48.834] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.834] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:21535b365ca8dc1c epochNumber=1 {"provingJobId":"1:BASE_PARITY:21535b365ca8dc1c"} 18:41:48 [18:41:48.834] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:efd5f7b0324775aa epochNumber=1 {"provingJobId":"1:BASE_PARITY:efd5f7b0324775aa"} 18:41:48 [18:41:48.835] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2782e96469609fef epochNumber=2 {"provingJobId":"1:BASE_PARITY:2782e96469609fef"} 18:41:48 [18:41:48.835] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7f3c3efd1fc30ac7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7f3c3efd1fc30ac7"} 18:41:48 [18:41:48.835] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.836] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:49c6ed039e7598bf epochNumber=1 {"provingJobId":"1:BASE_PARITY:49c6ed039e7598bf"} 18:41:48 [18:41:48.836] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4111a4acf8bba55e epochNumber=1 {"provingJobId":"1:BASE_PARITY:4111a4acf8bba55e"} 18:41:48 [18:41:48.836] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:96513ef80b69b956 epochNumber=2 {"provingJobId":"1:BASE_PARITY:96513ef80b69b956"} 18:41:48 [18:41:48.836] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:416adce9776cdbbf epochNumber=1 {"provingJobId":"1:BASE_PARITY:416adce9776cdbbf"} 18:41:48 [18:41:48.837] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.838] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5756a63817e74680 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5756a63817e74680"} 18:41:48 [18:41:48.838] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3fcaf2497ab08bbb epochNumber=1 {"provingJobId":"1:BASE_PARITY:3fcaf2497ab08bbb"} 18:41:48 [18:41:48.838] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a341216c460a3aa9 epochNumber=2 {"provingJobId":"1:BASE_PARITY:a341216c460a3aa9"} 18:41:48 [18:41:48.838] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:be2ed9ba40c96e62 epochNumber=1 {"provingJobId":"1:BASE_PARITY:be2ed9ba40c96e62"} 18:41:48 [18:41:48.839] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.840] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b97a76b3402144e2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b97a76b3402144e2"} 18:41:48 [18:41:48.840] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5dc81bd3ec81b2b7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5dc81bd3ec81b2b7"} 18:41:48 [18:41:48.840] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b97a76b3402144e2 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b97a76b3402144e2"} 18:41:48 [18:41:48.841] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.842] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3d4cfd875a6310e9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3d4cfd875a6310e9"} 18:41:48 [18:41:48.842] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b41c1dfa7a03192b epochNumber=1 {"provingJobId":"1:BASE_PARITY:b41c1dfa7a03192b"} 18:41:48 [18:41:48.842] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3d4cfd875a6310e9 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:3d4cfd875a6310e9"} 18:41:48 [18:41:48.843] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.843] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4f6c7d8ff7f4d90a epochNumber=1 {"provingJobId":"1:BASE_PARITY:4f6c7d8ff7f4d90a"} 18:41:48 [18:41:48.844] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c4331e3eaf37c940 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c4331e3eaf37c940"} 18:41:48 [18:41:48.844] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4f6c7d8ff7f4d90a type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:4f6c7d8ff7f4d90a"} 18:41:48 [18:41:48.845] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.845] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b64cd40df3e7dc8d epochNumber=1 {"provingJobId":"1:BASE_PARITY:b64cd40df3e7dc8d"} 18:41:48 [18:41:48.846] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b64cd40df3e7dc8d {"provingJobId":"1:BASE_PARITY:b64cd40df3e7dc8d"} 18:41:48 [18:41:48.846] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b64cd40df3e7dc8d type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:b64cd40df3e7dc8d"} 18:41:48 [18:41:48.846] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8bb40cc25e3e5a3d epochNumber=1 {"provingJobId":"1:BASE_PARITY:8bb40cc25e3e5a3d"} 18:41:48 [18:41:48.847] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b64cd40df3e7dc8d has already been completed {"provingJobId":"1:BASE_PARITY:b64cd40df3e7dc8d"} 18:41:48 [18:41:48.847] INFO: prover-client:proving-broker Proving Broker started 18:41:48 [18:41:48.848] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b35d617f66ab6e2d epochNumber=1 {"provingJobId":"1:BASE_PARITY:b35d617f66ab6e2d"} 18:41:48 [18:41:48.848] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2192b7f9da8dbe7b epochNumber=2 {"provingJobId":"1:BASE_PARITY:2192b7f9da8dbe7b"} 18:41:49 [18:41:49.901] INFO: prover-client:proving-broker Proving Broker started 18:41:49 [18:41:49.902] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b35d617f66ab6e2d settled=false {"provingJobId":"1:BASE_PARITY:b35d617f66ab6e2d","status":"pending"} 18:41:49 [18:41:49.903] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2192b7f9da8dbe7b settled=false {"provingJobId":"1:BASE_PARITY:2192b7f9da8dbe7b","status":"pending"} 18:41:49 [18:41:49.904] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:b35d617f66ab6e2d type=BASE_PARITY startedAt=1750099308848 older agent has taken job {"provingJobId":"1:BASE_PARITY:b35d617f66ab6e2d"} 18:41:49 [18:41:49.905] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b35d617f66ab6e2d type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:b35d617f66ab6e2d"} 18:41:49 [18:41:49.907] INFO: prover-client:proving-broker Proving Broker started 18:41:49 [18:41:49.908] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ceda0af5da14664 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8ceda0af5da14664"} 18:41:49 [18:41:49.909] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:15f10fffe2befa44 epochNumber=2 {"provingJobId":"1:BASE_PARITY:15f10fffe2befa44"} 18:41:50 [18:41:50.910] INFO: prover-client:proving-broker Proving Broker started 18:41:50 [18:41:50.914] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8ceda0af5da14664 settled=false {"provingJobId":"1:BASE_PARITY:8ceda0af5da14664","status":"pending"} 18:41:50 [18:41:50.915] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:15f10fffe2befa44 settled=false {"provingJobId":"1:BASE_PARITY:15f10fffe2befa44","status":"pending"} 18:41:50 [18:41:50.915] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8ceda0af5da14664 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:8ceda0af5da14664"} 18:41:50 [18:41:50.919] INFO: prover-client:proving-broker Proving Broker started 18:41:50 [18:41:50.921] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9cd9385608d8e29b epochNumber=1 {"provingJobId":"1:BASE_PARITY:9cd9385608d8e29b"} 18:41:50 [18:41:50.922] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1cf85f812091b09b epochNumber=2 {"provingJobId":"1:BASE_PARITY:1cf85f812091b09b"} 18:41:51 [18:41:51.923] INFO: prover-client:proving-broker Proving Broker started 18:41:51 [18:41:51.924] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9cd9385608d8e29b settled=false {"provingJobId":"1:BASE_PARITY:9cd9385608d8e29b","status":"pending"} 18:41:51 [18:41:51.925] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1cf85f812091b09b settled=false {"provingJobId":"1:BASE_PARITY:1cf85f812091b09b","status":"pending"} 18:41:51 [18:41:51.926] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9cd9385608d8e29b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9cd9385608d8e29b"} 18:41:51 [18:41:51.926] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9cd9385608d8e29b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:9cd9385608d8e29b"} 18:41:51 [18:41:51.929] INFO: prover-client:proving-broker Proving Broker started 18:41:51 [18:41:51.929] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3a558151f5340c73 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3a558151f5340c73"} 18:41:51 [18:41:51.930] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:04ec5ac794705e71 epochNumber=2 {"provingJobId":"1:BASE_PARITY:04ec5ac794705e71"} 18:41:51 [18:41:51.930] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3a558151f5340c73 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3a558151f5340c73"} 18:41:51 [18:41:51.931] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:04ec5ac794705e71 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:04ec5ac794705e71"} 18:41:51 [18:41:51.932] INFO: prover-client:proving-broker Proving Broker started 18:41:51 [18:41:51.932] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea721f55e9f0469b epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea721f55e9f0469b"} 18:41:51 [18:41:51.933] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6ec96c1e31864fe2 epochNumber=2 {"provingJobId":"1:BASE_PARITY:6ec96c1e31864fe2"} 18:41:51 [18:41:51.933] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ea721f55e9f0469b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ea721f55e9f0469b"} 18:41:51 [18:41:51.933] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ea721f55e9f0469b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ea721f55e9f0469b"} 18:41:51 [18:41:51.934] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6ec96c1e31864fe2 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:6ec96c1e31864fe2"} 18:41:51 [18:41:51.935] INFO: prover-client:proving-broker Proving Broker started 18:41:51 [18:41:51.935] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:069fce1fee138d36 err=test error {"provingJoId":"1:BASE_PARITY:069fce1fee138d36"} 18:41:51 [18:41:51.936] INFO: prover-client:proving-broker Proving Broker started 18:41:51 [18:41:51.937] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:43145cca648533ee not found {"provingJobId":"1:BASE_PARITY:43145cca648533ee"} 18:41:51 [18:41:51.938] INFO: prover-client:proving-broker Proving Broker started 18:41:51 [18:41:51.939] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ae03077d04881d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8ae03077d04881d6"} 18:41:51 [18:41:51.940] INFO: prover-client:proving-broker Proving Broker started 18:41:51 [18:41:51.941] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f3de5e929cde8e8a epochNumber=1 {"provingJobId":"1:BASE_PARITY:f3de5e929cde8e8a"} 18:41:52 [18:41:52.043] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f3de5e929cde8e8a timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:f3de5e929cde8e8a"} 18:41:52 [18:41:52.050] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.051] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:092a62ba99daa96a epochNumber=1 {"provingJobId":"1:BASE_PARITY:092a62ba99daa96a"} 18:41:52 [18:41:52.153] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:092a62ba99daa96a timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:092a62ba99daa96a"} 18:41:52 [18:41:52.160] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:80e5d8074ac201a7 epochNumber=10 {"provingJobId":"1:BASE_PARITY:80e5d8074ac201a7"} 18:41:52 [18:41:52.179] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:52 [18:41:52.288] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.289] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9778998ce894370b epochNumber=1 {"provingJobId":"1:BASE_PARITY:9778998ce894370b"} 18:41:52 [18:41:52.441] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9778998ce894370b timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:9778998ce894370b"} 18:41:52 [18:41:52.491] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.492] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0e589b380ae96b31 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0e589b380ae96b31"} 18:41:52 [18:41:52.493] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0e589b380ae96b31 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:0e589b380ae96b31"} 18:41:52 [18:41:52.495] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.495] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:935ddf5ad9bae069 epochNumber=1 {"provingJobId":"1:BASE_PARITY:935ddf5ad9bae069"} 18:41:52 [18:41:52.496] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:935ddf5ad9bae069 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:935ddf5ad9bae069"} 18:41:52 [18:41:52.497] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:935ddf5ad9bae069 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:935ddf5ad9bae069"} 18:41:52 [18:41:52.498] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.498] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ff94ce619b67214a epochNumber=1 {"provingJobId":"1:BASE_PARITY:ff94ce619b67214a"} 18:41:52 [18:41:52.498] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ff94ce619b67214a type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ff94ce619b67214a"} 18:41:52 [18:41:52.499] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.500] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4e0f8aa6b74958b3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4e0f8aa6b74958b3"} 18:41:52 [18:41:52.500] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4e0f8aa6b74958b3 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:4e0f8aa6b74958b3"} 18:41:52 [18:41:52.501] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b592bf26bb4513fc epochNumber=3 {"provingJobId":"1:BASE_PARITY:b592bf26bb4513fc"} 18:41:52 [18:41:52.525] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:52 [18:41:52.525] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4e0f8aa6b74958b3 err=test error {"provingJoId":"1:BASE_PARITY:4e0f8aa6b74958b3"} 18:41:52 [18:41:52.527] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.527] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:93336d8383229a4a settled=false {"provingJobId":"1:BASE_PARITY:93336d8383229a4a","status":"pending"} 18:41:52 [18:41:52.528] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:04671b434accb612 settled=false {"provingJobId":"1:BASE_PARITY:04671b434accb612","status":"pending"} 18:41:52 [18:41:52.529] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.530] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:59ff1593a9bc9e3a settled=true {"provingJobId":"1:BASE_PARITY:59ff1593a9bc9e3a","status":"fulfilled"} 18:41:52 [18:41:52.530] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:1849bb8b42cff467 settled=true {"provingJobId":"2:BASE_PARITY:1849bb8b42cff467","status":"fulfilled"} 18:41:52 [18:41:52.531] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.531] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5fa1ba1ead0e7245 settled=true {"provingJobId":"1:BASE_PARITY:5fa1ba1ead0e7245","status":"fulfilled"} 18:41:52 [18:41:52.531] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d36270a2b27cf953 settled=false {"provingJobId":"1:BASE_PARITY:d36270a2b27cf953","status":"pending"} 18:41:52 [18:41:52.532] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.533] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4109fbe6c6f56b4d epochNumber=1 {"provingJobId":"1:BASE_PARITY:4109fbe6c6f56b4d"} 18:41:52 [18:41:52.535] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.535] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9fa3d8dbff7faeb6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9fa3d8dbff7faeb6"} 18:41:52 [18:41:52.536] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:9fa3d8dbff7faeb6: Error: db error: Error: db error 18:41:52 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:9fa3d8dbff7faeb6"} 18:41:52 [18:41:52.537] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.537] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f19abbd2a2dcdfee epochNumber=1 {"provingJobId":"1:BASE_PARITY:f19abbd2a2dcdfee"} 18:41:52 [18:41:52.538] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f19abbd2a2dcdfee type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f19abbd2a2dcdfee"} 18:41:52 [18:41:52.538] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f19abbd2a2dcdfee type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f19abbd2a2dcdfee"} 18:41:52 [18:41:52.539] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.539] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae759ac756c0d636 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ae759ac756c0d636"} 18:41:52 [18:41:52.540] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ae759ac756c0d636 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ae759ac756c0d636"} 18:41:52 [18:41:52.540] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ae759ac756c0d636 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ae759ac756c0d636"} 18:41:52 [18:41:52.540] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:ae759ac756c0d636: Error: db error 18:41:52 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:ae759ac756c0d636"} 18:41:52 [18:41:52.541] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.542] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d2b058f8f0ab7c5b epochNumber=1 {"provingJobId":"1:BASE_PARITY:d2b058f8f0ab7c5b"} 18:41:52 [18:41:52.542] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d2b058f8f0ab7c5b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d2b058f8f0ab7c5b"} 18:41:52 [18:41:52.542] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d2b058f8f0ab7c5b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:d2b058f8f0ab7c5b"} 18:41:52 [18:41:52.543] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.544] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c26d3627c0e7eeb2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c26d3627c0e7eeb2"} 18:41:52 [18:41:52.545] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c26d3627c0e7eeb2 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c26d3627c0e7eeb2"} 18:41:52 [18:41:52.545] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c26d3627c0e7eeb2 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:c26d3627c0e7eeb2"} 18:41:52 [18:41:52.547] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:c26d3627c0e7eeb2 jobErr=test error: Error: db error 18:41:52 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:c26d3627c0e7eeb2"} 18:41:52 [18:41:52.549] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.550] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2765eed0045ad00e not found {"provingJobId":"1:BASE_PARITY:2765eed0045ad00e"} 18:41:52 [18:41:52.552] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.553] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ac8e97cfed66a8ff err=test error {"provingJoId":"1:BASE_PARITY:ac8e97cfed66a8ff"} 18:41:52 [18:41:52.554] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.565] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6a11482a0f027d3a epochNumber=1 {"provingJobId":"1:BASE_PARITY:6a11482a0f027d3a"} 18:41:52 [18:41:52.566] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6a11482a0f027d3a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:6a11482a0f027d3a"} 18:41:52 [18:41:52.566] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6a11482a0f027d3a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6a11482a0f027d3a"} 18:41:52 [18:41:52.577] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:6068f1a580fd81c7 epochNumber=2 {"provingJobId":"2:BASE_PARITY:6068f1a580fd81c7"} 18:41:52 [18:41:52.578] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:6068f1a580fd81c7 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:6068f1a580fd81c7"} 18:41:52 [18:41:52.578] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:6068f1a580fd81c7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:6068f1a580fd81c7"} 18:41:52 [18:41:52.640] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:7f379f8f1b53d841 epochNumber=3 {"provingJobId":"3:BASE_PARITY:7f379f8f1b53d841"} 18:41:52 [18:41:52.656] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:52 [18:41:52.703] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:f7b75b836950de9a epochNumber=4 {"provingJobId":"4:BASE_PARITY:f7b75b836950de9a"} 18:41:52 [18:41:52.708] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:52 [18:41:52.766] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:66de97b0859893a4 epochNumber=5 {"provingJobId":"5:BASE_PARITY:66de97b0859893a4"} 18:41:52 [18:41:52.785] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:52 [18:41:52.822] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.823] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:829d20d6a095649f epochNumber=42 {"provingJobId":"1:BASE_PARITY:829d20d6a095649f"} 18:41:52 [18:41:52.824] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testNgUskG/42 with map size 1073741824 18:41:52 [18:41:52.827] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:52 [18:41:52.853] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:43f808a8481286af epochNumber=1 {"provingJobId":"1:BASE_PARITY:43f808a8481286af"} 18:41:52 [18:41:52.859] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.860] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:73a65b5c6fe81dba epochNumber=1 {"provingJobId":"1:BASE_PARITY:73a65b5c6fe81dba"} 18:41:52 [18:41:52.860] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5Wgqth/1 with map size 1073741824 18:41:52 [18:41:52.865] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:52 [18:41:52.882] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:67af1088013cab30 epochNumber=1 {"provingJobId":"1:BASE_PARITY:67af1088013cab30"} 18:41:52 [18:41:52.891] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.891] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6884bd5d678f49d5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6884bd5d678f49d5"} 18:41:52 [18:41:52.893] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZCWVx8/1 with map size 1073741824 18:41:52 [18:41:52.895] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:52 [18:41:52.914] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6884bd5d678f49d5 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:6884bd5d678f49d5"} 18:41:52 [18:41:52.917] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.917] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3d56ce323769fdae epochNumber=1 {"provingJobId":"1:BASE_PARITY:3d56ce323769fdae"} 18:41:52 [18:41:52.918] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHXmnzM/1 with map size 1073741824 18:41:52 [18:41:52.919] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:52 [18:41:52.936] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3d56ce323769fdae epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:3d56ce323769fdae"} 18:41:52 [18:41:52.936] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3d56ce323769fdae type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3d56ce323769fdae"} 18:41:52 [18:41:52.943] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3d56ce323769fdae epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:3d56ce323769fdae"} 18:41:52 [18:41:52.946] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.946] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f5d8ddb7e7554903 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f5d8ddb7e7554903"} 18:41:52 [18:41:52.947] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdMRvj2/1 with map size 1073741824 18:41:52 [18:41:52.948] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:52 [18:41:52.971] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.973] INFO: prover-client:proving-broker Proving Broker started 18:41:52 [18:41:52.974] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:85109dcd761cfc61 epochNumber=1 {"provingJobId":"1:BASE_PARITY:85109dcd761cfc61"} 18:41:52 [18:41:52.974] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRRwbHI/1 with map size 1073741824 18:41:52 [18:41:52.976] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:52 [18:41:52.993] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:85109dcd761cfc61 {"provingJobId":"1:BASE_PARITY:85109dcd761cfc61"} 18:41:52 [18:41:52.993] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:85109dcd761cfc61 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:85109dcd761cfc61"} 18:41:52 [18:41:52.994] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:85109dcd761cfc61 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:85109dcd761cfc61"} 18:41:53 [18:41:53.002] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.003] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b2f6bd0fc63bc679 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b2f6bd0fc63bc679"} 18:41:53 [18:41:53.003] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhlEnHK/1 with map size 1073741824 18:41:53 [18:41:53.005] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.021] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b2f6bd0fc63bc679 {"provingJobId":"1:BASE_PARITY:b2f6bd0fc63bc679"} 18:41:53 [18:41:53.022] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b2f6bd0fc63bc679 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:b2f6bd0fc63bc679"} 18:41:53 [18:41:53.031] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.032] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d2df622223546ab0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d2df622223546ab0"} 18:41:53 [18:41:53.032] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test81jotS/1 with map size 1073741824 18:41:53 [18:41:53.033] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.049] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d2df622223546ab0 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d2df622223546ab0"} 18:41:53 [18:41:53.049] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d2df622223546ab0 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d2df622223546ab0"} 18:41:53 [18:41:53.058] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.058] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4ca42bb3726921fc epochNumber=1 {"provingJobId":"1:BASE_PARITY:4ca42bb3726921fc"} 18:41:53 [18:41:53.059] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvppQj4/1 with map size 1073741824 18:41:53 [18:41:53.061] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.089] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4ca42bb3726921fc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4ca42bb3726921fc"} 18:41:53 [18:41:53.091] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4ca42bb3726921fc type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4ca42bb3726921fc"} 18:41:53 [18:41:53.115] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.121] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:460ee3138b9ac3b1 epochNumber=0 {"provingJobId":"1:BASE_PARITY:460ee3138b9ac3b1"} 18:41:53 [18:41:53.124] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testMQTxnm/0 with map size 1073741824 18:41:53 [18:41:53.129] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.147] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:460ee3138b9ac3b1 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:460ee3138b9ac3b1"} 18:41:53 [18:41:53.148] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:998824750dacc0d9 epochNumber=0 {"provingJobId":"1:BASE_PARITY:998824750dacc0d9"} 18:41:53 [18:41:53.148] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:460ee3138b9ac3b1 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:460ee3138b9ac3b1"} 18:41:53 [18:41:53.149] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:792465d69323fee0 epochNumber=0 {"provingJobId":"1:BASE_PARITY:792465d69323fee0"} 18:41:53 [18:41:53.164] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.166] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.167] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:54aa4c68041d44dc epochNumber=2 {"provingJobId":"1:BASE_PARITY:54aa4c68041d44dc"} 18:41:53 [18:41:53.168] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testB49RjY/2 with map size 1073741824 18:41:53 [18:41:53.170] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.186] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:da6a87ff63e1bb99 epochNumber=1 {"provingJobId":"1:BASE_PARITY:da6a87ff63e1bb99"} 18:41:53 [18:41:53.187] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testB49RjY/1 with map size 1073741824 18:41:53 [18:41:53.189] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.224] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.225] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:286e66d8fa672eb8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:286e66d8fa672eb8"} 18:41:53 [18:41:53.226] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testp2CO7y/1 with map size 1073741824 18:41:53 [18:41:53.227] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.246] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.247] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:caa7b95af23bf6e0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:caa7b95af23bf6e0"} 18:41:53 [18:41:53.247] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWVEmNw/1 with map size 1073741824 18:41:53 [18:41:53.249] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.270] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3a44f59f840ae72c epochNumber=1 {"provingJobId":"1:BASE_PARITY:3a44f59f840ae72c"} 18:41:53 [18:41:53.277] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:19b51b234570bc20 epochNumber=2 {"provingJobId":"1:BASE_PARITY:19b51b234570bc20"} 18:41:53 [18:41:53.278] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWVEmNw/2 with map size 1073741824 18:41:53 [18:41:53.281] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.313] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:523b5732ec83f8a7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:523b5732ec83f8a7"} 18:41:53 [18:41:53.336] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.337] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:90dca3f56d81eb4c epochNumber=1 {"provingJobId":"1:BASE_PARITY:90dca3f56d81eb4c"} 18:41:53 [18:41:53.337] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testm3l3wj/1 with map size 1073741824 18:41:53 [18:41:53.340] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.361] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2f19495470b39ced epochNumber=1 {"provingJobId":"1:BASE_PARITY:2f19495470b39ced"} 18:41:53 [18:41:53.368] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:507ad71eda708e93 epochNumber=2 {"provingJobId":"1:BASE_PARITY:507ad71eda708e93"} 18:41:53 [18:41:53.369] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testm3l3wj/2 with map size 1073741824 18:41:53 [18:41:53.371] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.387] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:91a8d0d0c06027e8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:91a8d0d0c06027e8"} 18:41:53 [18:41:53.396] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.397] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3fd3a808586d4995 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3fd3a808586d4995"} 18:41:53 [18:41:53.398] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6T1vzm/1 with map size 1073741824 18:41:53 [18:41:53.417] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.440] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cb7e84a80cd9b288 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cb7e84a80cd9b288"} 18:41:53 [18:41:53.448] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9c6f13f69c0b7021 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9c6f13f69c0b7021"} 18:41:53 [18:41:53.449] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6T1vzm/2 with map size 1073741824 18:41:53 [18:41:53.450] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.470] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7c16760896aa8eb3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7c16760896aa8eb3"} 18:41:53 [18:41:53.481] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.481] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2207df8a47eb6926 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2207df8a47eb6926"} 18:41:53 [18:41:53.482] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgObJBZ/1 with map size 1073741824 18:41:53 [18:41:53.484] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.501] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c84d2dc072d81b15 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c84d2dc072d81b15"} 18:41:53 [18:41:53.511] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2207df8a47eb6926 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2207df8a47eb6926"} 18:41:53 [18:41:53.523] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.524] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dc86913a6415213f epochNumber=1 {"provingJobId":"1:BASE_PARITY:dc86913a6415213f"} 18:41:53 [18:41:53.525] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4UjEtc/1 with map size 1073741824 18:41:53 [18:41:53.529] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eccbf73122d87777 epochNumber=1 {"provingJobId":"1:BASE_PARITY:eccbf73122d87777"} 18:41:53 [18:41:53.552] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dc86913a6415213f type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:dc86913a6415213f"} 18:41:53 [18:41:53.562] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.563] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eb8fe4620cbf47a7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:eb8fe4620cbf47a7"} 18:41:53 [18:41:53.564] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH4mGse/1 with map size 1073741824 18:41:53 [18:41:53.566] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.585] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:78fc2bf6a52f71b2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:78fc2bf6a52f71b2"} 18:41:53 [18:41:53.591] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:eb8fe4620cbf47a7 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:eb8fe4620cbf47a7"} 18:41:53 [18:41:53.597] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.598] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:62dc6677f9e8fbb5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:62dc6677f9e8fbb5"} 18:41:53 [18:41:53.599] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlzmvXl/1 with map size 1073741824 18:41:53 [18:41:53.601] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.630] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:62dc6677f9e8fbb5 {"provingJobId":"1:BASE_PARITY:62dc6677f9e8fbb5"} 18:41:53 [18:41:53.631] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:62dc6677f9e8fbb5 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:62dc6677f9e8fbb5"} 18:41:53 [18:41:53.637] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:df845ea816268b59 epochNumber=1 {"provingJobId":"1:BASE_PARITY:df845ea816268b59"} 18:41:53 [18:41:53.646] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:62dc6677f9e8fbb5 has already been completed {"provingJobId":"1:BASE_PARITY:62dc6677f9e8fbb5"} 18:41:53 [18:41:53.650] INFO: prover-client:proving-broker Proving Broker started 18:41:53 [18:41:53.651] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9b8c150f1698e975 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9b8c150f1698e975"} 18:41:53 [18:41:53.654] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEu3WHh/1 with map size 1073741824 18:41:53 [18:41:53.658] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:53 [18:41:53.677] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:265c728a4d0d447b epochNumber=2 {"provingJobId":"1:BASE_PARITY:265c728a4d0d447b"} 18:41:53 [18:41:53.678] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testEu3WHh/2 with map size 1073741824 18:41:53 [18:41:53.680] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:54 [18:41:54.751] INFO: prover-client:proving-broker Proving Broker started 18:41:54 [18:41:54.756] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9b8c150f1698e975 settled=false {"provingJobId":"1:BASE_PARITY:9b8c150f1698e975","status":"pending"} 18:41:54 [18:41:54.757] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:265c728a4d0d447b settled=false {"provingJobId":"1:BASE_PARITY:265c728a4d0d447b","status":"pending"} 18:41:54 [18:41:54.758] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:9b8c150f1698e975 type=BASE_PARITY startedAt=1750099313699 older agent has taken job {"provingJobId":"1:BASE_PARITY:9b8c150f1698e975"} 18:41:54 [18:41:54.758] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9b8c150f1698e975 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:9b8c150f1698e975"} 18:41:54 [18:41:54.761] INFO: prover-client:proving-broker Proving Broker started 18:41:54 [18:41:54.762] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c7b52a09e64a6620 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c7b52a09e64a6620"} 18:41:54 [18:41:54.762] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testS3B5pW/1 with map size 1073741824 18:41:54 [18:41:54.764] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:54 [18:41:54.782] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f9651a0c5816a954 epochNumber=2 {"provingJobId":"1:BASE_PARITY:f9651a0c5816a954"} 18:41:54 [18:41:54.783] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testS3B5pW/2 with map size 1073741824 18:41:54 [18:41:54.785] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:55 [18:41:55.804] INFO: prover-client:proving-broker Proving Broker started 18:41:55 [18:41:55.807] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c7b52a09e64a6620 settled=false {"provingJobId":"1:BASE_PARITY:c7b52a09e64a6620","status":"pending"} 18:41:55 [18:41:55.810] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f9651a0c5816a954 settled=false {"provingJobId":"1:BASE_PARITY:f9651a0c5816a954","status":"pending"} 18:41:55 [18:41:55.811] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c7b52a09e64a6620 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:c7b52a09e64a6620"} 18:41:55 [18:41:55.817] INFO: prover-client:proving-broker Proving Broker started 18:41:55 [18:41:55.818] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aa9d2d0f14d5ae87 epochNumber=1 {"provingJobId":"1:BASE_PARITY:aa9d2d0f14d5ae87"} 18:41:55 [18:41:55.819] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8W6wvh/1 with map size 1073741824 18:41:55 [18:41:55.821] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:55 [18:41:55.841] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b920fa7867ca0502 epochNumber=2 {"provingJobId":"1:BASE_PARITY:b920fa7867ca0502"} 18:41:55 [18:41:55.843] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8W6wvh/2 with map size 1073741824 18:41:55 [18:41:55.846] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:56 [18:41:56.868] INFO: prover-client:proving-broker Proving Broker started 18:41:56 [18:41:56.870] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:aa9d2d0f14d5ae87 settled=false {"provingJobId":"1:BASE_PARITY:aa9d2d0f14d5ae87","status":"pending"} 18:41:56 [18:41:56.872] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b920fa7867ca0502 settled=false {"provingJobId":"1:BASE_PARITY:b920fa7867ca0502","status":"pending"} 18:41:56 [18:41:56.873] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:aa9d2d0f14d5ae87 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:aa9d2d0f14d5ae87"} 18:41:56 [18:41:56.873] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:aa9d2d0f14d5ae87 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:aa9d2d0f14d5ae87"} 18:41:56 [18:41:56.884] INFO: prover-client:proving-broker Proving Broker started 18:41:56 [18:41:56.885] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:49f75a5a66a1a639 epochNumber=1 {"provingJobId":"1:BASE_PARITY:49f75a5a66a1a639"} 18:41:56 [18:41:56.885] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWWOUeG/1 with map size 1073741824 18:41:56 [18:41:56.887] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:56 [18:41:56.903] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3cbfb62d0426a915 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3cbfb62d0426a915"} 18:41:56 [18:41:56.904] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWWOUeG/2 with map size 1073741824 18:41:56 [18:41:56.907] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:56 [18:41:56.926] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:49f75a5a66a1a639 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:49f75a5a66a1a639"} 18:41:56 [18:41:56.934] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3cbfb62d0426a915 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3cbfb62d0426a915"} 18:41:56 [18:41:56.945] INFO: prover-client:proving-broker Proving Broker started 18:41:56 [18:41:56.946] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a6ce951c10795579 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a6ce951c10795579"} 18:41:56 [18:41:56.947] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testT7TSQb/1 with map size 1073741824 18:41:56 [18:41:56.949] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:56 [18:41:56.965] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6cd23d4f93a99d08 epochNumber=2 {"provingJobId":"1:BASE_PARITY:6cd23d4f93a99d08"} 18:41:56 [18:41:56.966] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testT7TSQb/2 with map size 1073741824 18:41:56 [18:41:56.968] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:56 [18:41:56.983] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a6ce951c10795579 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a6ce951c10795579"} 18:41:56 [18:41:56.983] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a6ce951c10795579 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:a6ce951c10795579"} 18:41:56 [18:41:56.989] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6cd23d4f93a99d08 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:6cd23d4f93a99d08"} 18:41:56 [18:41:56.998] INFO: prover-client:proving-broker Proving Broker started 18:41:56 [18:41:56.999] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:9b32a79116edd623 err=test error {"provingJoId":"1:BASE_PARITY:9b32a79116edd623"} 18:41:57 [18:41:57.002] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.003] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0d1ace3e32e3f111 not found {"provingJobId":"1:BASE_PARITY:0d1ace3e32e3f111"} 18:41:57 [18:41:57.010] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.011] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:85a88281bd792010 epochNumber=1 {"provingJobId":"1:BASE_PARITY:85a88281bd792010"} 18:41:57 [18:41:57.012] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjauEmp/1 with map size 1073741824 18:41:57 [18:41:57.014] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.035] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.035] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8e8b72ec90025f36 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8e8b72ec90025f36"} 18:41:57 [18:41:57.036] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdcQ34A/1 with map size 1073741824 18:41:57 [18:41:57.039] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.164] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8e8b72ec90025f36 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:8e8b72ec90025f36"} 18:41:57 [18:41:57.173] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.174] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e1f23c2eaa69b547 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e1f23c2eaa69b547"} 18:41:57 [18:41:57.175] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiAanhh/1 with map size 1073741824 18:41:57 [18:41:57.177] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.299] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e1f23c2eaa69b547 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:e1f23c2eaa69b547"} 18:41:57 [18:41:57.305] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2896baf0ae7b3151 epochNumber=10 {"provingJobId":"1:BASE_PARITY:2896baf0ae7b3151"} 18:41:57 [18:41:57.306] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testiAanhh/10 with map size 1073741824 18:41:57 [18:41:57.310] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.349] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:57 [18:41:57.350] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 18:41:57 [18:41:57.462] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.463] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:453d61898fadd840 epochNumber=1 {"provingJobId":"1:BASE_PARITY:453d61898fadd840"} 18:41:57 [18:41:57.463] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQnqZM6/1 with map size 1073741824 18:41:57 [18:41:57.466] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.640] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:453d61898fadd840 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:453d61898fadd840"} 18:41:57 [18:41:57.690] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.691] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:259573412dc97503 epochNumber=1 {"provingJobId":"1:BASE_PARITY:259573412dc97503"} 18:41:57 [18:41:57.693] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZw3zIP/1 with map size 1073741824 18:41:57 [18:41:57.694] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.712] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:259573412dc97503 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:259573412dc97503"} 18:41:57 [18:41:57.717] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.718] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:af22a9e7e069eb76 epochNumber=1 {"provingJobId":"1:BASE_PARITY:af22a9e7e069eb76"} 18:41:57 [18:41:57.720] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testR5fji3/1 with map size 1073741824 18:41:57 [18:41:57.722] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.740] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:af22a9e7e069eb76 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:af22a9e7e069eb76"} 18:41:57 [18:41:57.741] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:af22a9e7e069eb76 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:af22a9e7e069eb76"} 18:41:57 [18:41:57.752] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.753] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:775394b098303902 epochNumber=1 {"provingJobId":"1:BASE_PARITY:775394b098303902"} 18:41:57 [18:41:57.754] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test16Vi2A/1 with map size 1073741824 18:41:57 [18:41:57.756] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.779] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:775394b098303902 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:775394b098303902"} 18:41:57 [18:41:57.794] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.795] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:182679b7963b3798 epochNumber=1 {"provingJobId":"1:BASE_PARITY:182679b7963b3798"} 18:41:57 [18:41:57.796] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0KsDb3/1 with map size 1073741824 18:41:57 [18:41:57.798] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.816] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:182679b7963b3798 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:182679b7963b3798"} 18:41:57 [18:41:57.818] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:176a6ecbd1b1a2f1 epochNumber=3 {"provingJobId":"1:BASE_PARITY:176a6ecbd1b1a2f1"} 18:41:57 [18:41:57.819] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test0KsDb3/3 with map size 1073741824 18:41:57 [18:41:57.821] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.844] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:57 [18:41:57.845] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 18:41:57 [18:41:57.865] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:182679b7963b3798 err=test error {"provingJoId":"1:BASE_PARITY:182679b7963b3798"} 18:41:57 [18:41:57.874] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7rojOm/1 with map size 1073741824 18:41:57 [18:41:57.877] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.902] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7rojOm/2 with map size 1073741824 18:41:57 [18:41:57.908] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.938] INFO: prover-client:proving-broker Proving Broker started 18:41:57 [18:41:57.940] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e2c704c18d554d13 settled=false {"provingJobId":"1:BASE_PARITY:e2c704c18d554d13","status":"pending"} 18:41:57 [18:41:57.943] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:efed55e08e71a873 settled=false {"provingJobId":"1:BASE_PARITY:efed55e08e71a873","status":"pending"} 18:41:57 [18:41:57.950] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testp7jBxZ/1 with map size 1073741824 18:41:57 [18:41:57.953] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:57 [18:41:57.968] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testp7jBxZ/2 with map size 1073741824 18:41:57 [18:41:57.970] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.003] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.005] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6107c9a0ca268121 settled=true {"provingJobId":"1:BASE_PARITY:6107c9a0ca268121","status":"fulfilled"} 18:41:58 [18:41:58.007] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:8dc55f9f1987a06c settled=true {"provingJobId":"2:BASE_PARITY:8dc55f9f1987a06c","status":"fulfilled"} 18:41:58 [18:41:58.011] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlYlXEA/1 with map size 1073741824 18:41:58 [18:41:58.015] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.047] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlYlXEA/2 with map size 1073741824 18:41:58 [18:41:58.052] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.076] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.077] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:561878b45a163856 settled=true {"provingJobId":"1:BASE_PARITY:561878b45a163856","status":"fulfilled"} 18:41:58 [18:41:58.079] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fe5d18d590ef206d settled=false {"provingJobId":"1:BASE_PARITY:fe5d18d590ef206d","status":"pending"} 18:41:58 [18:41:58.083] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.083] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1c9499b9e9de79d4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1c9499b9e9de79d4"} 18:41:58 [18:41:58.084] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhTERgk/1 with map size 1073741824 18:41:58 [18:41:58.085] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.110] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.111] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fcb753f83fc812d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:fcb753f83fc812d6"} 18:41:58 [18:41:58.112] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:fcb753f83fc812d6: Error: db error: Error: db error 18:41:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:fcb753f83fc812d6"} 18:41:58 [18:41:58.121] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.122] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5859ef3e2f8a17ab epochNumber=1 {"provingJobId":"1:BASE_PARITY:5859ef3e2f8a17ab"} 18:41:58 [18:41:58.122] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSW0pQr/1 with map size 1073741824 18:41:58 [18:41:58.125] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.142] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5859ef3e2f8a17ab type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5859ef3e2f8a17ab"} 18:41:58 [18:41:58.143] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5859ef3e2f8a17ab type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5859ef3e2f8a17ab"} 18:41:58 [18:41:58.155] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.156] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4070803dc51a0770 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4070803dc51a0770"} 18:41:58 [18:41:58.157] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnIVm7u/1 with map size 1073741824 18:41:58 [18:41:58.160] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.182] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4070803dc51a0770 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4070803dc51a0770"} 18:41:58 [18:41:58.182] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4070803dc51a0770 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:4070803dc51a0770"} 18:41:58 [18:41:58.183] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:4070803dc51a0770: Error: db error 18:41:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:4070803dc51a0770"} 18:41:58 [18:41:58.188] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.188] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d7efd07350881df6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d7efd07350881df6"} 18:41:58 [18:41:58.189] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaLlLXV/1 with map size 1073741824 18:41:58 [18:41:58.190] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.218] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d7efd07350881df6 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d7efd07350881df6"} 18:41:58 [18:41:58.219] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d7efd07350881df6 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:d7efd07350881df6"} 18:41:58 [18:41:58.230] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.231] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a27266ad295676cc epochNumber=1 {"provingJobId":"1:BASE_PARITY:a27266ad295676cc"} 18:41:58 [18:41:58.232] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXClJxH/1 with map size 1073741824 18:41:58 [18:41:58.236] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.259] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a27266ad295676cc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a27266ad295676cc"} 18:41:58 [18:41:58.260] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a27266ad295676cc type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:a27266ad295676cc"} 18:41:58 [18:41:58.261] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a27266ad295676cc jobErr=test error: Error: db error 18:41:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:a27266ad295676cc"} 18:41:58 [18:41:58.266] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.267] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:eca23d6cb6392cf8 not found {"provingJobId":"1:BASE_PARITY:eca23d6cb6392cf8"} 18:41:58 [18:41:58.272] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.272] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ad22f27b7dc8e419 err=test error {"provingJoId":"1:BASE_PARITY:ad22f27b7dc8e419"} 18:41:58 [18:41:58.278] INFO: prover-client:proving-broker Proving Broker started 18:41:58 [18:41:58.289] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1da5b5d590dae0e2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1da5b5d590dae0e2"} 18:41:58 [18:41:58.291] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdCe6wA/1 with map size 1073741824 18:41:58 [18:41:58.293] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.324] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1da5b5d590dae0e2 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:1da5b5d590dae0e2"} 18:41:58 [18:41:58.325] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1da5b5d590dae0e2 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:1da5b5d590dae0e2"} 18:41:58 [18:41:58.343] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:83a9f23f8e49873f epochNumber=2 {"provingJobId":"2:BASE_PARITY:83a9f23f8e49873f"} 18:41:58 [18:41:58.346] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testdCe6wA/2 with map size 1073741824 18:41:58 [18:41:58.350] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.373] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:83a9f23f8e49873f type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:83a9f23f8e49873f"} 18:41:58 [18:41:58.373] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:83a9f23f8e49873f type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:83a9f23f8e49873f"} 18:41:58 [18:41:58.443] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:a694ebf2186dc4eb epochNumber=3 {"provingJobId":"3:BASE_PARITY:a694ebf2186dc4eb"} 18:41:58 [18:41:58.446] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testdCe6wA/3 with map size 1073741824 18:41:58 [18:41:58.450] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.484] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:58 [18:41:58.485] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 18:41:58 [18:41:58.532] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:c3fc7e4ea7605b30 epochNumber=4 {"provingJobId":"4:BASE_PARITY:c3fc7e4ea7605b30"} 18:41:58 [18:41:58.533] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testdCe6wA/4 with map size 1073741824 18:41:58 [18:41:58.537] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.564] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:58 [18:41:58.564] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 18:41:58 [18:41:58.619] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:14e47ba41a23c21f epochNumber=5 {"provingJobId":"5:BASE_PARITY:14e47ba41a23c21f"} 18:41:58 [18:41:58.621] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testdCe6wA/5 with map size 1073741824 18:41:58 [18:41:58.625] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 18:41:58 [18:41:58.671] INFO: prover-client:proving-broker Cleaned up jobs=1 18:41:58 [18:41:58.672] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 18:41:58 PASS src/proving_broker/proving_broker.test.ts (13.596 s) 18:41:58 ProvingBroker 18:41:58 Producer API 18:41:58 ✓ refuses stale jobs (39 ms) 18:41:58 ✓ enqueues jobs (3 ms) 18:41:58 ✓ ignores duplicate jobs (4 ms) 18:41:58 ✓ reports correct status when enqueuing repeat jobs (5 ms) 18:41:58 ✓ throws an error in case of duplicate job IDs (27 ms) 18:41:58 ✓ returns not-found status for non-existing jobs (1 ms) 18:41:58 ✓ cancels jobs in queue (2 ms) 18:41:58 ✓ cancels jobs in-progress (2 ms) 18:41:58 ✓ returns job result if successful (2 ms) 18:41:58 ✓ returns job error if failed (2 ms) 18:41:58 ✓ correctly returns job status for concurrent writes (2 ms) 18:41:58 ✓ refuses stale jobs (37 ms) 18:41:58 ✓ enqueues jobs (32 ms) 18:41:58 ✓ ignores duplicate jobs (26 ms) 18:41:58 ✓ reports correct status when enqueuing repeat jobs (28 ms) 18:41:58 ✓ throws an error in case of duplicate job IDs (25 ms) 18:41:58 ✓ returns not-found status for non-existing jobs (3 ms) 18:41:58 ✓ cancels jobs in queue (28 ms) 18:41:58 ✓ cancels jobs in-progress (28 ms) 18:41:58 ✓ returns job result if successful (27 ms) 18:41:58 ✓ returns job error if failed (50 ms) 18:41:58 ✓ correctly returns job status for concurrent writes (54 ms) 18:41:58 Consumer API 18:41:58 ✓ returns undefined if no jobs are available (1 ms) 18:41:58 ✓ returns jobs in priority order (2 ms) 18:41:58 ✓ returns undefined if no jobs are available for the given allowList 18:41:58 ✓ returns a job if it is in the allowList (1 ms) 18:41:58 ✓ returns the most important job if it is in the allowList (2 ms) 18:41:58 ✓ returns any job if filter is empty (2 ms) 18:41:58 ✓ returns a new job when reporting job success (2 ms) 18:41:58 ✓ returns a new job when reporting permanent error (2 ms) 18:41:58 ✓ returns a new job when reporting retry-able error (2 ms) 18:41:58 ✓ returns a new job when reporting progress if current one is cancelled (2 ms) 18:41:58 ✓ returns a new job if job is already in progress elsewhere (1059 ms) 18:41:58 ✓ avoids sending the same job to a new agent after a restart (1012 ms) 18:41:58 ✓ avoids sending a completed job to a new agent after a restart (1010 ms) 18:41:58 ✓ tracks job result if in progress (3 ms) 18:41:58 ✓ tracks job result even if job is in queue (2 ms) 18:41:58 ✓ ignores reported job error if unknown job (2 ms) 18:41:58 ✓ ignores job result if unknown job (1 ms) 18:41:58 ✓ returns undefined if no jobs are available (2 ms) 18:41:58 ✓ returns jobs in priority order (54 ms) 18:41:58 ✓ returns undefined if no jobs are available for the given allowList (25 ms) 18:41:58 ✓ returns a job if it is in the allowList (89 ms) 18:41:58 ✓ returns the most important job if it is in the allowList (59 ms) 18:41:58 ✓ returns any job if filter is empty (85 ms) 18:41:58 ✓ returns a new job when reporting job success (42 ms) 18:41:58 ✓ returns a new job when reporting permanent error (38 ms) 18:41:58 ✓ returns a new job when reporting retry-able error (35 ms) 18:41:58 ✓ returns a new job when reporting progress if current one is cancelled (53 ms) 18:41:58 ✓ returns a new job if job is already in progress elsewhere (1111 ms) 18:41:58 ✓ avoids sending the same job to a new agent after a restart (1053 ms) 18:41:58 ✓ avoids sending a completed job to a new agent after a restart (1068 ms) 18:41:58 ✓ tracks job result if in progress (61 ms) 18:41:58 ✓ tracks job result even if job is in queue (53 ms) 18:41:58 ✓ ignores reported job error if unknown job (3 ms) 18:41:58 ✓ ignores job result if unknown job (4 ms) 18:41:58 Timeouts 18:41:58 ✓ tracks in progress jobs (2 ms) 18:41:58 ✓ re-enqueues jobs that time out (109 ms) 18:41:58 ✓ cancel stale jobs that time out (237 ms) 18:41:58 ✓ keeps the jobs in progress while it is alive (203 ms) 18:41:58 ✓ tracks in progress jobs (27 ms) 18:41:58 ✓ re-enqueues jobs that time out (138 ms) 18:41:58 ✓ cancel stale jobs that time out (285 ms) 18:41:58 ✓ keeps the jobs in progress while it is alive (231 ms) 18:41:58 Retries 18:41:58 ✓ retries jobs (3 ms) 18:41:58 ✓ retries up to a maximum number of times (3 ms) 18:41:58 ✓ passing retry=false does not retry (2 ms) 18:41:58 ✓ does not retry if job is stale (27 ms) 18:41:58 ✓ retries jobs (26 ms) 18:41:58 ✓ retries up to a maximum number of times (34 ms) 18:41:58 ✓ passing retry=false does not retry (37 ms) 18:41:58 ✓ does not retry if job is stale (80 ms) 18:41:58 Database management 18:41:58 ✓ re-enqueues proof requests on start (3 ms) 18:41:58 ✓ restores proof results on start (1 ms) 18:41:58 ✓ only re-enqueues unfinished jobs (1 ms) 18:41:58 ✓ saves job when enqueued (2 ms) 18:41:58 ✓ does not retain job if database fails to save (3 ms) 18:41:58 ✓ saves job result (2 ms) 18:41:58 ✓ saves result even if database fails to save (2 ms) 18:41:58 ✓ saves job error (2 ms) 18:41:58 ✓ saves job error even if database fails to save (5 ms) 18:41:58 ✓ does not save job result if job is unknown (3 ms) 18:41:58 ✓ does not save job error if job is unknown (2 ms) 18:41:58 ✓ cleans up old jobs periodically (265 ms) 18:41:58 ✓ re-enqueues proof requests on start (78 ms) 18:41:58 ✓ restores proof results on start (61 ms) 18:41:58 ✓ only re-enqueues unfinished jobs (72 ms) 18:41:58 ✓ saves job when enqueued (26 ms) 18:41:58 ✓ does not retain job if database fails to save (5 ms) 18:41:58 ✓ saves job result (37 ms) 18:41:58 ✓ saves result even if database fails to save (32 ms) 18:41:58 ✓ saves job error (40 ms) 18:41:58 ✓ saves job error even if database fails to save (37 ms) 18:41:58 ✓ does not save job result if job is unknown (5 ms) 18:41:58 ✓ does not save job error if job is unknown (5 ms) 18:41:58 ✓ cleans up old jobs periodically (426 ms) 18:41:58 18:41:58 Test Suites: 1 passed, 1 total 18:41:58 Tests: 96 passed, 96 total 18:41:58 Snapshots: 0 total 18:41:58 Time: 13.727 s 18:41:58 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 18:41:58 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?