ACI ·

Parent Log: http://ci.aztec-labs.com/81043fcb565088fe Command: bcc565b00bd8bb22 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/bcb41f0bbab679d1b5d50e6b59b4af0faeeb9b48 Env: REF_NAME=gh-readonly-queue/next/pr-15138-3c671e0822075208f705135771771f5a47e9652e CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Wed Jun 18 16:17:41 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15138_amd64_x3-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 16:17:45 [16:17:45.466] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.472] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a53b3d914470854c epochNumber=42 {"provingJobId":"1:BASE_PARITY:a53b3d914470854c"} 16:17:45 [16:17:45.476] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:c4155967dbd1ec90 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c4155967dbd1ec90"} 16:17:45 [16:17:45.505] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.506] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a47b4a32cad23ee5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a47b4a32cad23ee5"} 16:17:45 [16:17:45.507] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea11ad30e86276e8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea11ad30e86276e8"} 16:17:45 [16:17:45.509] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.510] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cc8f408a2fbbf9d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cc8f408a2fbbf9d6"} 16:17:45 [16:17:45.512] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cc8f408a2fbbf9d6 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:cc8f408a2fbbf9d6"} 16:17:45 [16:17:45.514] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.515] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:26a1b8e939b4513d epochNumber=1 {"provingJobId":"1:BASE_PARITY:26a1b8e939b4513d"} 16:17:45 [16:17:45.516] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:26a1b8e939b4513d epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:26a1b8e939b4513d"} 16:17:45 [16:17:45.516] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:26a1b8e939b4513d type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:26a1b8e939b4513d"} 16:17:45 [16:17:45.517] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:26a1b8e939b4513d epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:26a1b8e939b4513d"} 16:17:45 [16:17:45.519] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.519] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8b76311f810cb78f epochNumber=1 {"provingJobId":"1:BASE_PARITY:8b76311f810cb78f"} 16:17:45 [16:17:45.548] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.549] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.550] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3a67b350abfd5bed epochNumber=1 {"provingJobId":"1:BASE_PARITY:3a67b350abfd5bed"} 16:17:45 [16:17:45.550] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3a67b350abfd5bed {"provingJobId":"1:BASE_PARITY:3a67b350abfd5bed"} 16:17:45 [16:17:45.551] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3a67b350abfd5bed type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3a67b350abfd5bed"} 16:17:45 [16:17:45.551] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3a67b350abfd5bed type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:3a67b350abfd5bed"} 16:17:45 [16:17:45.553] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.553] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c66c8f5622780d21 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c66c8f5622780d21"} 16:17:45 [16:17:45.554] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c66c8f5622780d21 {"provingJobId":"1:BASE_PARITY:c66c8f5622780d21"} 16:17:45 [16:17:45.554] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c66c8f5622780d21 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:c66c8f5622780d21"} 16:17:45 [16:17:45.556] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.556] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:efc5c62dac552da6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:efc5c62dac552da6"} 16:17:45 [16:17:45.557] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:efc5c62dac552da6 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:efc5c62dac552da6"} 16:17:45 [16:17:45.557] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:efc5c62dac552da6 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:efc5c62dac552da6"} 16:17:45 [16:17:45.558] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.559] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8341f15ff3326a46 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8341f15ff3326a46"} 16:17:45 [16:17:45.559] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8341f15ff3326a46 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8341f15ff3326a46"} 16:17:45 [16:17:45.559] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8341f15ff3326a46 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:8341f15ff3326a46"} 16:17:45 [16:17:45.560] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.561] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ffea671ff951e6e epochNumber=0 {"provingJobId":"1:BASE_PARITY:8ffea671ff951e6e"} 16:17:45 [16:17:45.561] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8ffea671ff951e6e epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8ffea671ff951e6e"} 16:17:45 [16:17:45.561] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a59d22c6fcf5a1c0 epochNumber=0 {"provingJobId":"1:BASE_PARITY:a59d22c6fcf5a1c0"} 16:17:45 [16:17:45.561] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8ffea671ff951e6e epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8ffea671ff951e6e"} 16:17:45 [16:17:45.561] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8360aa57cba5dec8 epochNumber=0 {"provingJobId":"1:BASE_PARITY:8360aa57cba5dec8"} 16:17:45 [16:17:45.562] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.570] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.570] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2a41d2696b576bbb epochNumber=2 {"provingJobId":"1:BASE_PARITY:2a41d2696b576bbb"} 16:17:45 [16:17:45.571] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4c2eef49b3eec93e epochNumber=1 {"provingJobId":"1:BASE_PARITY:4c2eef49b3eec93e"} 16:17:45 [16:17:45.572] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.572] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1b14f08e7e241027 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1b14f08e7e241027"} 16:17:45 [16:17:45.573] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.573] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f415acfd4087d632 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f415acfd4087d632"} 16:17:45 [16:17:45.574] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:54584b6846dbec86 epochNumber=1 {"provingJobId":"1:BASE_PARITY:54584b6846dbec86"} 16:17:45 [16:17:45.574] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b8edca5c2c5d23ad epochNumber=2 {"provingJobId":"1:BASE_PARITY:b8edca5c2c5d23ad"} 16:17:45 [16:17:45.574] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5e0f8afd0d9e1d5d epochNumber=1 {"provingJobId":"1:BASE_PARITY:5e0f8afd0d9e1d5d"} 16:17:45 [16:17:45.575] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.575] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:839bc587ad01c7ce epochNumber=1 {"provingJobId":"1:BASE_PARITY:839bc587ad01c7ce"} 16:17:45 [16:17:45.575] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:39168461c455f3a3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:39168461c455f3a3"} 16:17:45 [16:17:45.576] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4a424781f3603040 epochNumber=2 {"provingJobId":"1:BASE_PARITY:4a424781f3603040"} 16:17:45 [16:17:45.576] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7d516f3e4d0ce08b epochNumber=1 {"provingJobId":"1:BASE_PARITY:7d516f3e4d0ce08b"} 16:17:45 [16:17:45.577] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.578] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c9db57dd4245ae21 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c9db57dd4245ae21"} 16:17:45 [16:17:45.578] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5cbcfb856d1b7333 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5cbcfb856d1b7333"} 16:17:45 [16:17:45.578] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:532674d4301200e0 epochNumber=2 {"provingJobId":"1:BASE_PARITY:532674d4301200e0"} 16:17:45 [16:17:45.579] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cad9896dc3ea4b09 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cad9896dc3ea4b09"} 16:17:45 [16:17:45.580] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.580] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:649f0fc418f293e6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:649f0fc418f293e6"} 16:17:45 [16:17:45.581] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6361f661e6c1a09b epochNumber=1 {"provingJobId":"1:BASE_PARITY:6361f661e6c1a09b"} 16:17:45 [16:17:45.581] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:649f0fc418f293e6 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:649f0fc418f293e6"} 16:17:45 [16:17:45.582] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.582] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a922041be18046be epochNumber=1 {"provingJobId":"1:BASE_PARITY:a922041be18046be"} 16:17:45 [16:17:45.583] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fa2c4c782f97b7ae epochNumber=1 {"provingJobId":"1:BASE_PARITY:fa2c4c782f97b7ae"} 16:17:45 [16:17:45.583] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a922041be18046be type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:a922041be18046be"} 16:17:45 [16:17:45.584] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.584] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eb9362d92807d889 epochNumber=1 {"provingJobId":"1:BASE_PARITY:eb9362d92807d889"} 16:17:45 [16:17:45.585] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e70fd25783f0da2d epochNumber=1 {"provingJobId":"1:BASE_PARITY:e70fd25783f0da2d"} 16:17:45 [16:17:45.585] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:eb9362d92807d889 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:eb9362d92807d889"} 16:17:45 [16:17:45.586] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.587] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c52425b0f11931bc epochNumber=1 {"provingJobId":"1:BASE_PARITY:c52425b0f11931bc"} 16:17:45 [16:17:45.587] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c52425b0f11931bc {"provingJobId":"1:BASE_PARITY:c52425b0f11931bc"} 16:17:45 [16:17:45.587] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c52425b0f11931bc type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:c52425b0f11931bc"} 16:17:45 [16:17:45.588] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5d0deab148965e56 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5d0deab148965e56"} 16:17:45 [16:17:45.588] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c52425b0f11931bc has already been completed {"provingJobId":"1:BASE_PARITY:c52425b0f11931bc"} 16:17:45 [16:17:45.589] INFO: prover-client:proving-broker Proving Broker started 16:17:45 [16:17:45.589] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3baeb3ef2d7b6ead epochNumber=1 {"provingJobId":"1:BASE_PARITY:3baeb3ef2d7b6ead"} 16:17:45 [16:17:45.590] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:016a644ecc0723ed epochNumber=2 {"provingJobId":"1:BASE_PARITY:016a644ecc0723ed"} 16:17:46 [16:17:46.642] INFO: prover-client:proving-broker Proving Broker started 16:17:46 [16:17:46.643] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3baeb3ef2d7b6ead settled=false {"provingJobId":"1:BASE_PARITY:3baeb3ef2d7b6ead","status":"pending"} 16:17:46 [16:17:46.645] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:016a644ecc0723ed settled=false {"provingJobId":"1:BASE_PARITY:016a644ecc0723ed","status":"pending"} 16:17:46 [16:17:46.646] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:3baeb3ef2d7b6ead type=BASE_PARITY startedAt=1750263465590 older agent has taken job {"provingJobId":"1:BASE_PARITY:3baeb3ef2d7b6ead"} 16:17:46 [16:17:46.647] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3baeb3ef2d7b6ead type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:3baeb3ef2d7b6ead"} 16:17:46 [16:17:46.648] INFO: prover-client:proving-broker Proving Broker started 16:17:46 [16:17:46.649] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a57a65e19125c2fc epochNumber=1 {"provingJobId":"1:BASE_PARITY:a57a65e19125c2fc"} 16:17:46 [16:17:46.649] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:197d5aa18d57c238 epochNumber=2 {"provingJobId":"1:BASE_PARITY:197d5aa18d57c238"} 16:17:47 [16:17:47.652] INFO: prover-client:proving-broker Proving Broker started 16:17:47 [16:17:47.653] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a57a65e19125c2fc settled=false {"provingJobId":"1:BASE_PARITY:a57a65e19125c2fc","status":"pending"} 16:17:47 [16:17:47.654] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:197d5aa18d57c238 settled=false {"provingJobId":"1:BASE_PARITY:197d5aa18d57c238","status":"pending"} 16:17:47 [16:17:47.656] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a57a65e19125c2fc type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:a57a65e19125c2fc"} 16:17:47 [16:17:47.659] INFO: prover-client:proving-broker Proving Broker started 16:17:47 [16:17:47.660] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:74e4feb59dfebc26 epochNumber=1 {"provingJobId":"1:BASE_PARITY:74e4feb59dfebc26"} 16:17:47 [16:17:47.661] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4b42fa68a511837f epochNumber=2 {"provingJobId":"1:BASE_PARITY:4b42fa68a511837f"} 16:17:48 [16:17:48.662] INFO: prover-client:proving-broker Proving Broker started 16:17:48 [16:17:48.663] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:74e4feb59dfebc26 settled=false {"provingJobId":"1:BASE_PARITY:74e4feb59dfebc26","status":"pending"} 16:17:48 [16:17:48.663] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4b42fa68a511837f settled=false {"provingJobId":"1:BASE_PARITY:4b42fa68a511837f","status":"pending"} 16:17:48 [16:17:48.663] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:74e4feb59dfebc26 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:74e4feb59dfebc26"} 16:17:48 [16:17:48.664] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:74e4feb59dfebc26 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:74e4feb59dfebc26"} 16:17:48 [16:17:48.665] INFO: prover-client:proving-broker Proving Broker started 16:17:48 [16:17:48.665] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f076da07c89fb05b epochNumber=1 {"provingJobId":"1:BASE_PARITY:f076da07c89fb05b"} 16:17:48 [16:17:48.666] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4e3fa24f49a2a1cb epochNumber=2 {"provingJobId":"1:BASE_PARITY:4e3fa24f49a2a1cb"} 16:17:48 [16:17:48.666] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f076da07c89fb05b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f076da07c89fb05b"} 16:17:48 [16:17:48.666] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4e3fa24f49a2a1cb type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4e3fa24f49a2a1cb"} 16:17:48 [16:17:48.667] INFO: prover-client:proving-broker Proving Broker started 16:17:48 [16:17:48.668] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a6e7ae1876ea5b4c epochNumber=1 {"provingJobId":"1:BASE_PARITY:a6e7ae1876ea5b4c"} 16:17:48 [16:17:48.668] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:671323b70b8c64d6 epochNumber=2 {"provingJobId":"1:BASE_PARITY:671323b70b8c64d6"} 16:17:48 [16:17:48.668] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a6e7ae1876ea5b4c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a6e7ae1876ea5b4c"} 16:17:48 [16:17:48.668] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a6e7ae1876ea5b4c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:a6e7ae1876ea5b4c"} 16:17:48 [16:17:48.669] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:671323b70b8c64d6 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:671323b70b8c64d6"} 16:17:48 [16:17:48.669] INFO: prover-client:proving-broker Proving Broker started 16:17:48 [16:17:48.670] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b9000c5a1e6676e4 err=test error {"provingJoId":"1:BASE_PARITY:b9000c5a1e6676e4"} 16:17:48 [16:17:48.670] INFO: prover-client:proving-broker Proving Broker started 16:17:48 [16:17:48.671] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:dc897d2646d4d2bd not found {"provingJobId":"1:BASE_PARITY:dc897d2646d4d2bd"} 16:17:48 [16:17:48.672] INFO: prover-client:proving-broker Proving Broker started 16:17:48 [16:17:48.672] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7c15ac5058f5c2df epochNumber=1 {"provingJobId":"1:BASE_PARITY:7c15ac5058f5c2df"} 16:17:48 [16:17:48.673] INFO: prover-client:proving-broker Proving Broker started 16:17:48 [16:17:48.673] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6bed0a237332ac28 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6bed0a237332ac28"} 16:17:48 [16:17:48.774] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6bed0a237332ac28 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:6bed0a237332ac28"} 16:17:48 [16:17:48.782] INFO: prover-client:proving-broker Proving Broker started 16:17:48 [16:17:48.782] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6adeebe26e817b69 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6adeebe26e817b69"} 16:17:48 [16:17:48.884] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6adeebe26e817b69 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:6adeebe26e817b69"} 16:17:48 [16:17:48.891] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f8d491b7ea830469 epochNumber=10 {"provingJobId":"1:BASE_PARITY:f8d491b7ea830469"} 16:17:48 [16:17:48.910] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:49 [16:17:49.019] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.020] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1adb641fac3877cf epochNumber=1 {"provingJobId":"1:BASE_PARITY:1adb641fac3877cf"} 16:17:49 [16:17:49.173] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1adb641fac3877cf timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:1adb641fac3877cf"} 16:17:49 [16:17:49.222] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.223] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:91e23be3986f5850 epochNumber=1 {"provingJobId":"1:BASE_PARITY:91e23be3986f5850"} 16:17:49 [16:17:49.225] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:91e23be3986f5850 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:91e23be3986f5850"} 16:17:49 [16:17:49.227] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.228] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:878515f3be64c5bd epochNumber=1 {"provingJobId":"1:BASE_PARITY:878515f3be64c5bd"} 16:17:49 [16:17:49.229] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:878515f3be64c5bd type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:878515f3be64c5bd"} 16:17:49 [16:17:49.230] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:878515f3be64c5bd type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:878515f3be64c5bd"} 16:17:49 [16:17:49.232] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.232] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4c64a78af9c247ef epochNumber=1 {"provingJobId":"1:BASE_PARITY:4c64a78af9c247ef"} 16:17:49 [16:17:49.233] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4c64a78af9c247ef type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4c64a78af9c247ef"} 16:17:49 [16:17:49.235] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.235] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:169b9f56f9cfe6b4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:169b9f56f9cfe6b4"} 16:17:49 [16:17:49.236] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:169b9f56f9cfe6b4 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:169b9f56f9cfe6b4"} 16:17:49 [16:17:49.237] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8de657ea2338723d epochNumber=3 {"provingJobId":"1:BASE_PARITY:8de657ea2338723d"} 16:17:49 [16:17:49.261] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:49 [16:17:49.263] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:169b9f56f9cfe6b4 err=test error {"provingJoId":"1:BASE_PARITY:169b9f56f9cfe6b4"} 16:17:49 [16:17:49.266] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.267] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cfb4acab89df2601 settled=false {"provingJobId":"1:BASE_PARITY:cfb4acab89df2601","status":"pending"} 16:17:49 [16:17:49.267] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4d40881ac206bf14 settled=false {"provingJobId":"1:BASE_PARITY:4d40881ac206bf14","status":"pending"} 16:17:49 [16:17:49.271] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.271] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:210823a90a025f4c settled=true {"provingJobId":"1:BASE_PARITY:210823a90a025f4c","status":"fulfilled"} 16:17:49 [16:17:49.271] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:bf34efd0d93cc4b9 settled=true {"provingJobId":"2:BASE_PARITY:bf34efd0d93cc4b9","status":"fulfilled"} 16:17:49 [16:17:49.273] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.274] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5757519f1f70797c settled=true {"provingJobId":"1:BASE_PARITY:5757519f1f70797c","status":"fulfilled"} 16:17:49 [16:17:49.274] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8f81f12ae1b54af1 settled=false {"provingJobId":"1:BASE_PARITY:8f81f12ae1b54af1","status":"pending"} 16:17:49 [16:17:49.277] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.279] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:590ae95302ab1b96 epochNumber=1 {"provingJobId":"1:BASE_PARITY:590ae95302ab1b96"} 16:17:49 [16:17:49.282] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.282] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f3551a37a25b9471 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f3551a37a25b9471"} 16:17:49 [16:17:49.284] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:f3551a37a25b9471: Error: db error: Error: db error 16:17:49 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:f3551a37a25b9471"} 16:17:49 [16:17:49.286] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.287] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7cde4dfd9d3256b7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7cde4dfd9d3256b7"} 16:17:49 [16:17:49.287] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7cde4dfd9d3256b7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:7cde4dfd9d3256b7"} 16:17:49 [16:17:49.288] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7cde4dfd9d3256b7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:7cde4dfd9d3256b7"} 16:17:49 [16:17:49.289] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.290] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4b96b8f24fb12a79 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4b96b8f24fb12a79"} 16:17:49 [16:17:49.290] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4b96b8f24fb12a79 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4b96b8f24fb12a79"} 16:17:49 [16:17:49.291] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4b96b8f24fb12a79 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:4b96b8f24fb12a79"} 16:17:49 [16:17:49.292] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:4b96b8f24fb12a79: Error: db error 16:17:49 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:4b96b8f24fb12a79"} 16:17:49 [16:17:49.293] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.294] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4abd2cf8a07d497c epochNumber=1 {"provingJobId":"1:BASE_PARITY:4abd2cf8a07d497c"} 16:17:49 [16:17:49.294] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4abd2cf8a07d497c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4abd2cf8a07d497c"} 16:17:49 [16:17:49.295] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4abd2cf8a07d497c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4abd2cf8a07d497c"} 16:17:49 [16:17:49.296] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.296] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e077b3ac9b1f84b8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e077b3ac9b1f84b8"} 16:17:49 [16:17:49.296] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e077b3ac9b1f84b8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e077b3ac9b1f84b8"} 16:17:49 [16:17:49.297] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e077b3ac9b1f84b8 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:e077b3ac9b1f84b8"} 16:17:49 [16:17:49.297] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:e077b3ac9b1f84b8 jobErr=test error: Error: db error 16:17:49 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:e077b3ac9b1f84b8"} 16:17:49 [16:17:49.298] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.298] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:987d9699d69e5beb not found {"provingJobId":"1:BASE_PARITY:987d9699d69e5beb"} 16:17:49 [16:17:49.299] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.300] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a1e87419ee83d76d err=test error {"provingJoId":"1:BASE_PARITY:a1e87419ee83d76d"} 16:17:49 [16:17:49.300] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.311] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bff0a0e246a5a73b epochNumber=1 {"provingJobId":"1:BASE_PARITY:bff0a0e246a5a73b"} 16:17:49 [16:17:49.312] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bff0a0e246a5a73b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:bff0a0e246a5a73b"} 16:17:49 [16:17:49.312] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bff0a0e246a5a73b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:bff0a0e246a5a73b"} 16:17:49 [16:17:49.322] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:88cfffec63d035e9 epochNumber=2 {"provingJobId":"2:BASE_PARITY:88cfffec63d035e9"} 16:17:49 [16:17:49.322] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:88cfffec63d035e9 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:88cfffec63d035e9"} 16:17:49 [16:17:49.323] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:88cfffec63d035e9 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:88cfffec63d035e9"} 16:17:49 [16:17:49.384] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:e100a671fde2bdf5 epochNumber=3 {"provingJobId":"3:BASE_PARITY:e100a671fde2bdf5"} 16:17:49 [16:17:49.402] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:49 [16:17:49.449] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:f7febd9d54883d8b epochNumber=4 {"provingJobId":"4:BASE_PARITY:f7febd9d54883d8b"} 16:17:49 [16:17:49.457] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:49 [16:17:49.513] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:f245c69629bd3d51 epochNumber=5 {"provingJobId":"5:BASE_PARITY:f245c69629bd3d51"} 16:17:49 [16:17:49.533] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:49 [16:17:49.569] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.569] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4baba62333b602d8 epochNumber=42 {"provingJobId":"1:BASE_PARITY:4baba62333b602d8"} 16:17:49 [16:17:49.571] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testRohKcH/42 with map size 1073741824 16:17:49 [16:17:49.575] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.601] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:15ffb9ed4c463171 epochNumber=1 {"provingJobId":"1:BASE_PARITY:15ffb9ed4c463171"} 16:17:49 [16:17:49.606] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.607] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a8d23e3ed99ce9ce epochNumber=1 {"provingJobId":"1:BASE_PARITY:a8d23e3ed99ce9ce"} 16:17:49 [16:17:49.608] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testo1LwsU/1 with map size 1073741824 16:17:49 [16:17:49.610] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.629] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:84cf2f5df80cc9af epochNumber=1 {"provingJobId":"1:BASE_PARITY:84cf2f5df80cc9af"} 16:17:49 [16:17:49.639] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.640] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea330d6191f3bd78 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea330d6191f3bd78"} 16:17:49 [16:17:49.640] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrjrJlf/1 with map size 1073741824 16:17:49 [16:17:49.645] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.664] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ea330d6191f3bd78 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:ea330d6191f3bd78"} 16:17:49 [16:17:49.669] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.670] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4a88f95042466217 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4a88f95042466217"} 16:17:49 [16:17:49.671] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2tQ3al/1 with map size 1073741824 16:17:49 [16:17:49.675] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.693] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4a88f95042466217 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:4a88f95042466217"} 16:17:49 [16:17:49.694] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4a88f95042466217 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:4a88f95042466217"} 16:17:49 [16:17:49.700] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4a88f95042466217 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:4a88f95042466217"} 16:17:49 [16:17:49.703] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.704] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:99cfe1d26c70b531 epochNumber=1 {"provingJobId":"1:BASE_PARITY:99cfe1d26c70b531"} 16:17:49 [16:17:49.705] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4NvXOg/1 with map size 1073741824 16:17:49 [16:17:49.707] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.732] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.734] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.735] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3ed3d46a4798ff14 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3ed3d46a4798ff14"} 16:17:49 [16:17:49.736] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrDj12h/1 with map size 1073741824 16:17:49 [16:17:49.738] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.758] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3ed3d46a4798ff14 {"provingJobId":"1:BASE_PARITY:3ed3d46a4798ff14"} 16:17:49 [16:17:49.759] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3ed3d46a4798ff14 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3ed3d46a4798ff14"} 16:17:49 [16:17:49.759] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3ed3d46a4798ff14 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:3ed3d46a4798ff14"} 16:17:49 [16:17:49.771] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.771] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c11a0e819ecbecb8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c11a0e819ecbecb8"} 16:17:49 [16:17:49.772] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testD9Z8wH/1 with map size 1073741824 16:17:49 [16:17:49.775] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.794] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c11a0e819ecbecb8 {"provingJobId":"1:BASE_PARITY:c11a0e819ecbecb8"} 16:17:49 [16:17:49.794] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c11a0e819ecbecb8 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:c11a0e819ecbecb8"} 16:17:49 [16:17:49.805] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.806] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:df44f3ca22b331b7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:df44f3ca22b331b7"} 16:17:49 [16:17:49.807] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTPi2XY/1 with map size 1073741824 16:17:49 [16:17:49.809] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.827] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:df44f3ca22b331b7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:df44f3ca22b331b7"} 16:17:49 [16:17:49.827] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:df44f3ca22b331b7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:df44f3ca22b331b7"} 16:17:49 [16:17:49.838] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.839] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:897cdacc945904b6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:897cdacc945904b6"} 16:17:49 [16:17:49.840] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTozxYa/1 with map size 1073741824 16:17:49 [16:17:49.843] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.860] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:897cdacc945904b6 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:897cdacc945904b6"} 16:17:49 [16:17:49.861] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:897cdacc945904b6 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:897cdacc945904b6"} 16:17:49 [16:17:49.870] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.871] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8fbb6c5f6dc5bac7 epochNumber=0 {"provingJobId":"1:BASE_PARITY:8fbb6c5f6dc5bac7"} 16:17:49 [16:17:49.871] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testszrNuA/0 with map size 1073741824 16:17:49 [16:17:49.874] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.893] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8fbb6c5f6dc5bac7 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8fbb6c5f6dc5bac7"} 16:17:49 [16:17:49.894] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6743d90c88286169 epochNumber=0 {"provingJobId":"1:BASE_PARITY:6743d90c88286169"} 16:17:49 [16:17:49.895] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8fbb6c5f6dc5bac7 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8fbb6c5f6dc5bac7"} 16:17:49 [16:17:49.895] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2a612ef591aab7fc epochNumber=0 {"provingJobId":"1:BASE_PARITY:2a612ef591aab7fc"} 16:17:49 [16:17:49.911] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.914] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.914] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bd9b20a16dccd19d epochNumber=2 {"provingJobId":"1:BASE_PARITY:bd9b20a16dccd19d"} 16:17:49 [16:17:49.915] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVpV7bC/2 with map size 1073741824 16:17:49 [16:17:49.917] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.935] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4baa6a7eba0bbe6f epochNumber=1 {"provingJobId":"1:BASE_PARITY:4baa6a7eba0bbe6f"} 16:17:49 [16:17:49.936] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVpV7bC/1 with map size 1073741824 16:17:49 [16:17:49.938] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.957] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.958] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5387aad970e10506 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5387aad970e10506"} 16:17:49 [16:17:49.958] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjHTaCw/1 with map size 1073741824 16:17:49 [16:17:49.960] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.978] INFO: prover-client:proving-broker Proving Broker started 16:17:49 [16:17:49.979] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03412f574a7558dd epochNumber=1 {"provingJobId":"1:BASE_PARITY:03412f574a7558dd"} 16:17:49 [16:17:49.979] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSjS4Q5/1 with map size 1073741824 16:17:49 [16:17:49.981] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:49 [16:17:49.997] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:39658134edca379b epochNumber=1 {"provingJobId":"1:BASE_PARITY:39658134edca379b"} 16:17:50 [16:17:50.003] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b762ae240012a04e epochNumber=2 {"provingJobId":"1:BASE_PARITY:b762ae240012a04e"} 16:17:50 [16:17:50.004] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSjS4Q5/2 with map size 1073741824 16:17:50 [16:17:50.006] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.024] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d7fb2a871d0c61a2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d7fb2a871d0c61a2"} 16:17:50 [16:17:50.041] INFO: prover-client:proving-broker Proving Broker started 16:17:50 [16:17:50.042] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:033e02cdf7a66a0c epochNumber=1 {"provingJobId":"1:BASE_PARITY:033e02cdf7a66a0c"} 16:17:50 [16:17:50.043] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMc0JRP/1 with map size 1073741824 16:17:50 [16:17:50.046] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.062] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9fb971cdfa36c130 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9fb971cdfa36c130"} 16:17:50 [16:17:50.068] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fd5885f758fef5b8 epochNumber=2 {"provingJobId":"1:BASE_PARITY:fd5885f758fef5b8"} 16:17:50 [16:17:50.069] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMc0JRP/2 with map size 1073741824 16:17:50 [16:17:50.072] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.089] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:120b0a9a604a7923 epochNumber=1 {"provingJobId":"1:BASE_PARITY:120b0a9a604a7923"} 16:17:50 [16:17:50.100] INFO: prover-client:proving-broker Proving Broker started 16:17:50 [16:17:50.101] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fc0d6e1e0cb97f2e epochNumber=1 {"provingJobId":"1:BASE_PARITY:fc0d6e1e0cb97f2e"} 16:17:50 [16:17:50.102] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0OoON8/1 with map size 1073741824 16:17:50 [16:17:50.104] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.121] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:48a9ecbdfdcc31b5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:48a9ecbdfdcc31b5"} 16:17:50 [16:17:50.127] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:015c7e740b8f3019 epochNumber=2 {"provingJobId":"1:BASE_PARITY:015c7e740b8f3019"} 16:17:50 [16:17:50.128] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0OoON8/2 with map size 1073741824 16:17:50 [16:17:50.130] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.147] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a0c718646213b893 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a0c718646213b893"} 16:17:50 [16:17:50.157] INFO: prover-client:proving-broker Proving Broker started 16:17:50 [16:17:50.157] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3a6f6e996e08b197 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3a6f6e996e08b197"} 16:17:50 [16:17:50.158] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeP6VI4/1 with map size 1073741824 16:17:50 [16:17:50.159] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.176] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:33ee26ba10d1b4cf epochNumber=1 {"provingJobId":"1:BASE_PARITY:33ee26ba10d1b4cf"} 16:17:50 [16:17:50.182] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3a6f6e996e08b197 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3a6f6e996e08b197"} 16:17:50 [16:17:50.194] INFO: prover-client:proving-broker Proving Broker started 16:17:50 [16:17:50.195] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b0674fbce52121e5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b0674fbce52121e5"} 16:17:50 [16:17:50.196] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCxkkgB/1 with map size 1073741824 16:17:50 [16:17:50.197] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.216] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b1a8eb20539e7415 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b1a8eb20539e7415"} 16:17:50 [16:17:50.223] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b0674fbce52121e5 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:b0674fbce52121e5"} 16:17:50 [16:17:50.232] INFO: prover-client:proving-broker Proving Broker started 16:17:50 [16:17:50.233] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cf3731a172186c34 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cf3731a172186c34"} 16:17:50 [16:17:50.233] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLaZxqP/1 with map size 1073741824 16:17:50 [16:17:50.235] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.251] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a59d57a78c57f83f epochNumber=1 {"provingJobId":"1:BASE_PARITY:a59d57a78c57f83f"} 16:17:50 [16:17:50.256] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cf3731a172186c34 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:cf3731a172186c34"} 16:17:50 [16:17:50.260] INFO: prover-client:proving-broker Proving Broker started 16:17:50 [16:17:50.260] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:485527d280292188 epochNumber=1 {"provingJobId":"1:BASE_PARITY:485527d280292188"} 16:17:50 [16:17:50.261] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZwJ2mM/1 with map size 1073741824 16:17:50 [16:17:50.262] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.278] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:485527d280292188 {"provingJobId":"1:BASE_PARITY:485527d280292188"} 16:17:50 [16:17:50.278] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:485527d280292188 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:485527d280292188"} 16:17:50 [16:17:50.285] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1f76bf3f45186942 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1f76bf3f45186942"} 16:17:50 [16:17:50.290] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:485527d280292188 has already been completed {"provingJobId":"1:BASE_PARITY:485527d280292188"} 16:17:50 [16:17:50.296] INFO: prover-client:proving-broker Proving Broker started 16:17:50 [16:17:50.297] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:da994e9a4b186bc9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:da994e9a4b186bc9"} 16:17:50 [16:17:50.297] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFXR3Y6/1 with map size 1073741824 16:17:50 [16:17:50.300] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:50 [16:17:50.317] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4d1377161104e661 epochNumber=2 {"provingJobId":"1:BASE_PARITY:4d1377161104e661"} 16:17:50 [16:17:50.318] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFXR3Y6/2 with map size 1073741824 16:17:50 [16:17:50.321] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:51 [16:17:51.391] INFO: prover-client:proving-broker Proving Broker started 16:17:51 [16:17:51.397] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:da994e9a4b186bc9 settled=false {"provingJobId":"1:BASE_PARITY:da994e9a4b186bc9","status":"pending"} 16:17:51 [16:17:51.399] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4d1377161104e661 settled=false {"provingJobId":"1:BASE_PARITY:4d1377161104e661","status":"pending"} 16:17:51 [16:17:51.401] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:da994e9a4b186bc9 type=BASE_PARITY startedAt=1750263470339 older agent has taken job {"provingJobId":"1:BASE_PARITY:da994e9a4b186bc9"} 16:17:51 [16:17:51.401] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:da994e9a4b186bc9 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:da994e9a4b186bc9"} 16:17:51 [16:17:51.406] INFO: prover-client:proving-broker Proving Broker started 16:17:51 [16:17:51.407] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fdd6f2f4a18f824f epochNumber=1 {"provingJobId":"1:BASE_PARITY:fdd6f2f4a18f824f"} 16:17:51 [16:17:51.411] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test58yLZy/1 with map size 1073741824 16:17:51 [16:17:51.414] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:51 [16:17:51.437] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5fec0afb2fa21695 epochNumber=2 {"provingJobId":"1:BASE_PARITY:5fec0afb2fa21695"} 16:17:51 [16:17:51.439] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test58yLZy/2 with map size 1073741824 16:17:51 [16:17:51.441] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:52 [16:17:52.461] INFO: prover-client:proving-broker Proving Broker started 16:17:52 [16:17:52.463] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fdd6f2f4a18f824f settled=false {"provingJobId":"1:BASE_PARITY:fdd6f2f4a18f824f","status":"pending"} 16:17:52 [16:17:52.464] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5fec0afb2fa21695 settled=false {"provingJobId":"1:BASE_PARITY:5fec0afb2fa21695","status":"pending"} 16:17:52 [16:17:52.465] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:fdd6f2f4a18f824f type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:fdd6f2f4a18f824f"} 16:17:52 [16:17:52.469] INFO: prover-client:proving-broker Proving Broker started 16:17:52 [16:17:52.469] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1b505d2ef6149ee8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1b505d2ef6149ee8"} 16:17:52 [16:17:52.470] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJahZ3o/1 with map size 1073741824 16:17:52 [16:17:52.471] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:52 [16:17:52.488] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2b74297eef10df7c epochNumber=2 {"provingJobId":"1:BASE_PARITY:2b74297eef10df7c"} 16:17:52 [16:17:52.489] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJahZ3o/2 with map size 1073741824 16:17:52 [16:17:52.491] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:53 [16:17:53.507] INFO: prover-client:proving-broker Proving Broker started 16:17:53 [16:17:53.510] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1b505d2ef6149ee8 settled=false {"provingJobId":"1:BASE_PARITY:1b505d2ef6149ee8","status":"pending"} 16:17:53 [16:17:53.512] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2b74297eef10df7c settled=false {"provingJobId":"1:BASE_PARITY:2b74297eef10df7c","status":"pending"} 16:17:53 [16:17:53.514] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1b505d2ef6149ee8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:1b505d2ef6149ee8"} 16:17:53 [16:17:53.514] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1b505d2ef6149ee8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:1b505d2ef6149ee8"} 16:17:53 [16:17:53.527] INFO: prover-client:proving-broker Proving Broker started 16:17:53 [16:17:53.528] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:99ca2877a9c668a3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:99ca2877a9c668a3"} 16:17:53 [16:17:53.529] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaKN4q4/1 with map size 1073741824 16:17:53 [16:17:53.531] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:53 [16:17:53.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:687786122d571039 epochNumber=2 {"provingJobId":"1:BASE_PARITY:687786122d571039"} 16:17:53 [16:17:53.547] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testaKN4q4/2 with map size 1073741824 16:17:53 [16:17:53.549] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:53 [16:17:53.566] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:99ca2877a9c668a3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:99ca2877a9c668a3"} 16:17:53 [16:17:53.572] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:687786122d571039 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:687786122d571039"} 16:17:53 [16:17:53.581] INFO: prover-client:proving-broker Proving Broker started 16:17:53 [16:17:53.582] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3332d095058375b1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3332d095058375b1"} 16:17:53 [16:17:53.583] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyRqwzP/1 with map size 1073741824 16:17:53 [16:17:53.584] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:53 [16:17:53.602] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4f5ec27afb24776b epochNumber=2 {"provingJobId":"1:BASE_PARITY:4f5ec27afb24776b"} 16:17:53 [16:17:53.603] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyRqwzP/2 with map size 1073741824 16:17:53 [16:17:53.610] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:53 [16:17:53.631] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3332d095058375b1 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3332d095058375b1"} 16:17:53 [16:17:53.632] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3332d095058375b1 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3332d095058375b1"} 16:17:53 [16:17:53.639] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4f5ec27afb24776b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4f5ec27afb24776b"} 16:17:53 [16:17:53.649] INFO: prover-client:proving-broker Proving Broker started 16:17:53 [16:17:53.650] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a600c67188a4cece err=test error {"provingJoId":"1:BASE_PARITY:a600c67188a4cece"} 16:17:53 [16:17:53.652] INFO: prover-client:proving-broker Proving Broker started 16:17:53 [16:17:53.653] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:08e3b5f44f4d5d02 not found {"provingJobId":"1:BASE_PARITY:08e3b5f44f4d5d02"} 16:17:53 [16:17:53.655] INFO: prover-client:proving-broker Proving Broker started 16:17:53 [16:17:53.656] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9bb95b54db1b5de8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9bb95b54db1b5de8"} 16:17:53 [16:17:53.658] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNnaOA1/1 with map size 1073741824 16:17:53 [16:17:53.660] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:53 [16:17:53.681] INFO: prover-client:proving-broker Proving Broker started 16:17:53 [16:17:53.682] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d6126d2b5bc4a876 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d6126d2b5bc4a876"} 16:17:53 [16:17:53.683] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOjQCAF/1 with map size 1073741824 16:17:53 [16:17:53.685] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:53 [16:17:53.809] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d6126d2b5bc4a876 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:d6126d2b5bc4a876"} 16:17:53 [16:17:53.818] INFO: prover-client:proving-broker Proving Broker started 16:17:53 [16:17:53.819] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3e87eafb8c412399 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3e87eafb8c412399"} 16:17:53 [16:17:53.821] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJoQaSC/1 with map size 1073741824 16:17:53 [16:17:53.823] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:53 [16:17:53.970] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3e87eafb8c412399 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:3e87eafb8c412399"} 16:17:53 [16:17:53.974] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:efa284e1d5933b0e epochNumber=10 {"provingJobId":"1:BASE_PARITY:efa284e1d5933b0e"} 16:17:53 [16:17:53.976] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testJoQaSC/10 with map size 1073741824 16:17:53 [16:17:53.978] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.021] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:54 [16:17:54.022] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:17:54 [16:17:54.131] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.132] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d3f425424dadb08c epochNumber=1 {"provingJobId":"1:BASE_PARITY:d3f425424dadb08c"} 16:17:54 [16:17:54.133] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHP0ITD/1 with map size 1073741824 16:17:54 [16:17:54.140] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.335] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d3f425424dadb08c timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:d3f425424dadb08c"} 16:17:54 [16:17:54.373] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.374] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ce11080f0c5bf5ab epochNumber=1 {"provingJobId":"1:BASE_PARITY:ce11080f0c5bf5ab"} 16:17:54 [16:17:54.375] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testo0AA80/1 with map size 1073741824 16:17:54 [16:17:54.377] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.396] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ce11080f0c5bf5ab type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:ce11080f0c5bf5ab"} 16:17:54 [16:17:54.402] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.403] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:420537c5a10b6c6c epochNumber=1 {"provingJobId":"1:BASE_PARITY:420537c5a10b6c6c"} 16:17:54 [16:17:54.405] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH2AvWt/1 with map size 1073741824 16:17:54 [16:17:54.408] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.428] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:420537c5a10b6c6c type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:420537c5a10b6c6c"} 16:17:54 [16:17:54.429] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:420537c5a10b6c6c type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:420537c5a10b6c6c"} 16:17:54 [16:17:54.450] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.452] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dd29c0339e34af98 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dd29c0339e34af98"} 16:17:54 [16:17:54.453] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyBxY35/1 with map size 1073741824 16:17:54 [16:17:54.456] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.475] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dd29c0339e34af98 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:dd29c0339e34af98"} 16:17:54 [16:17:54.484] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.485] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e7358ad55a1d9578 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e7358ad55a1d9578"} 16:17:54 [16:17:54.486] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testx9NWtj/1 with map size 1073741824 16:17:54 [16:17:54.487] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.507] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e7358ad55a1d9578 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:e7358ad55a1d9578"} 16:17:54 [16:17:54.509] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:af40ca0d6ff9b171 epochNumber=3 {"provingJobId":"1:BASE_PARITY:af40ca0d6ff9b171"} 16:17:54 [16:17:54.510] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testx9NWtj/3 with map size 1073741824 16:17:54 [16:17:54.514] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.538] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:54 [16:17:54.539] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:17:54 [16:17:54.558] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e7358ad55a1d9578 err=test error {"provingJoId":"1:BASE_PARITY:e7358ad55a1d9578"} 16:17:54 [16:17:54.572] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNHxykM/1 with map size 1073741824 16:17:54 [16:17:54.575] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.598] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNHxykM/2 with map size 1073741824 16:17:54 [16:17:54.601] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.622] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.625] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6e5e3e3571ba0faa settled=false {"provingJobId":"1:BASE_PARITY:6e5e3e3571ba0faa","status":"pending"} 16:17:54 [16:17:54.627] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f3684f81caab9ce3 settled=false {"provingJobId":"1:BASE_PARITY:f3684f81caab9ce3","status":"pending"} 16:17:54 [16:17:54.636] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7A2uhs/1 with map size 1073741824 16:17:54 [16:17:54.645] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.669] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7A2uhs/2 with map size 1073741824 16:17:54 [16:17:54.674] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.711] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.715] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:138673027b554f30 settled=true {"provingJobId":"1:BASE_PARITY:138673027b554f30","status":"fulfilled"} 16:17:54 [16:17:54.717] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:6d80aca6477c6469 settled=true {"provingJobId":"2:BASE_PARITY:6d80aca6477c6469","status":"fulfilled"} 16:17:54 [16:17:54.728] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLFeE1B/1 with map size 1073741824 16:17:54 [16:17:54.731] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.769] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testLFeE1B/2 with map size 1073741824 16:17:54 [16:17:54.773] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.794] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.797] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c6b746a0b9cc5521 settled=true {"provingJobId":"1:BASE_PARITY:c6b746a0b9cc5521","status":"fulfilled"} 16:17:54 [16:17:54.800] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8136553e5b36c969 settled=false {"provingJobId":"1:BASE_PARITY:8136553e5b36c969","status":"pending"} 16:17:54 [16:17:54.810] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.810] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:63802fd7b6dcbf2b epochNumber=1 {"provingJobId":"1:BASE_PARITY:63802fd7b6dcbf2b"} 16:17:54 [16:17:54.811] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBr6Rb4/1 with map size 1073741824 16:17:54 [16:17:54.814] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.845] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.845] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d9bcff18a5e0a218 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d9bcff18a5e0a218"} 16:17:54 [16:17:54.846] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:d9bcff18a5e0a218: Error: db error: Error: db error 16:17:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:d9bcff18a5e0a218"} 16:17:54 [16:17:54.853] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.854] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f602026c3976734f epochNumber=1 {"provingJobId":"1:BASE_PARITY:f602026c3976734f"} 16:17:54 [16:17:54.856] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6HFgUZ/1 with map size 1073741824 16:17:54 [16:17:54.862] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.881] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f602026c3976734f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f602026c3976734f"} 16:17:54 [16:17:54.882] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f602026c3976734f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f602026c3976734f"} 16:17:54 [16:17:54.897] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.897] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cda22d311ccf36da epochNumber=1 {"provingJobId":"1:BASE_PARITY:cda22d311ccf36da"} 16:17:54 [16:17:54.899] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPIsydz/1 with map size 1073741824 16:17:54 [16:17:54.905] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.926] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cda22d311ccf36da type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cda22d311ccf36da"} 16:17:54 [16:17:54.927] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cda22d311ccf36da type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cda22d311ccf36da"} 16:17:54 [16:17:54.928] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:cda22d311ccf36da: Error: db error 16:17:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:cda22d311ccf36da"} 16:17:54 [16:17:54.933] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.934] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6a5a3517715f0067 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6a5a3517715f0067"} 16:17:54 [16:17:54.935] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEsQErR/1 with map size 1073741824 16:17:54 [16:17:54.951] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:54 [16:17:54.969] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6a5a3517715f0067 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:6a5a3517715f0067"} 16:17:54 [16:17:54.970] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6a5a3517715f0067 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:6a5a3517715f0067"} 16:17:54 [16:17:54.982] INFO: prover-client:proving-broker Proving Broker started 16:17:54 [16:17:54.983] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dda1f8bbbbb19fa6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dda1f8bbbbb19fa6"} 16:17:54 [16:17:54.986] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLDX3Ck/1 with map size 1073741824 16:17:54 [16:17:54.990] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:55 [16:17:55.012] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:dda1f8bbbbb19fa6 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:dda1f8bbbbb19fa6"} 16:17:55 [16:17:55.013] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dda1f8bbbbb19fa6 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:dda1f8bbbbb19fa6"} 16:17:55 [16:17:55.015] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:dda1f8bbbbb19fa6 jobErr=test error: Error: db error 16:17:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:dda1f8bbbbb19fa6"} 16:17:55 [16:17:55.021] INFO: prover-client:proving-broker Proving Broker started 16:17:55 [16:17:55.023] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2a9bebc863f35325 not found {"provingJobId":"1:BASE_PARITY:2a9bebc863f35325"} 16:17:55 [16:17:55.028] INFO: prover-client:proving-broker Proving Broker started 16:17:55 [16:17:55.028] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:fd22c5a01c03c526 err=test error {"provingJoId":"1:BASE_PARITY:fd22c5a01c03c526"} 16:17:55 [16:17:55.033] INFO: prover-client:proving-broker Proving Broker started 16:17:55 [16:17:55.048] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:914c8cb66b087d87 epochNumber=1 {"provingJobId":"1:BASE_PARITY:914c8cb66b087d87"} 16:17:55 [16:17:55.049] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0CTHLE/1 with map size 1073741824 16:17:55 [16:17:55.056] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:55 [16:17:55.074] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:914c8cb66b087d87 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:914c8cb66b087d87"} 16:17:55 [16:17:55.075] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:914c8cb66b087d87 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:914c8cb66b087d87"} 16:17:55 [16:17:55.093] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:892d8da0bd28e1e3 epochNumber=2 {"provingJobId":"2:BASE_PARITY:892d8da0bd28e1e3"} 16:17:55 [16:17:55.095] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0CTHLE/2 with map size 1073741824 16:17:55 [16:17:55.099] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:55 [16:17:55.125] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:892d8da0bd28e1e3 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:892d8da0bd28e1e3"} 16:17:55 [16:17:55.126] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:892d8da0bd28e1e3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:892d8da0bd28e1e3"} 16:17:55 [16:17:55.195] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:370a997317925454 epochNumber=3 {"provingJobId":"3:BASE_PARITY:370a997317925454"} 16:17:55 [16:17:55.197] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test0CTHLE/3 with map size 1073741824 16:17:55 [16:17:55.201] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:55 [16:17:55.238] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:55 [16:17:55.238] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:17:55 [16:17:55.286] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:52644446be3ea5b6 epochNumber=4 {"provingJobId":"4:BASE_PARITY:52644446be3ea5b6"} 16:17:55 [16:17:55.287] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-test0CTHLE/4 with map size 1073741824 16:17:55 [16:17:55.291] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:55 [16:17:55.320] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:55 [16:17:55.320] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 16:17:55 [16:17:55.379] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:da1c02b3e39d8d7d epochNumber=5 {"provingJobId":"5:BASE_PARITY:da1c02b3e39d8d7d"} 16:17:55 [16:17:55.381] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-test0CTHLE/5 with map size 1073741824 16:17:55 [16:17:55.385] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:17:55 [16:17:55.425] INFO: prover-client:proving-broker Cleaned up jobs=1 16:17:55 [16:17:55.426] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 16:17:55 PASS src/proving_broker/proving_broker.test.ts (13.302 s) 16:17:55 ProvingBroker 16:17:55 Producer API 16:17:55 ✓ refuses stale jobs (41 ms) 16:17:55 ✓ enqueues jobs (4 ms) 16:17:55 ✓ ignores duplicate jobs (4 ms) 16:17:55 ✓ reports correct status when enqueuing repeat jobs (4 ms) 16:17:55 ✓ throws an error in case of duplicate job IDs (28 ms) 16:17:55 ✓ returns not-found status for non-existing jobs (2 ms) 16:17:55 ✓ cancels jobs in queue (3 ms) 16:17:55 ✓ cancels jobs in-progress (2 ms) 16:17:55 ✓ returns job result if successful (3 ms) 16:17:55 ✓ returns job error if failed (2 ms) 16:17:55 ✓ correctly returns job status for concurrent writes (2 ms) 16:17:55 ✓ refuses stale jobs (38 ms) 16:17:55 ✓ enqueues jobs (32 ms) 16:17:55 ✓ ignores duplicate jobs (30 ms) 16:17:55 ✓ reports correct status when enqueuing repeat jobs (33 ms) 16:17:55 ✓ throws an error in case of duplicate job IDs (27 ms) 16:17:55 ✓ returns not-found status for non-existing jobs (3 ms) 16:17:55 ✓ cancels jobs in queue (35 ms) 16:17:55 ✓ cancels jobs in-progress (33 ms) 16:17:55 ✓ returns job result if successful (32 ms) 16:17:55 ✓ returns job error if failed (33 ms) 16:17:55 ✓ correctly returns job status for concurrent writes (41 ms) 16:17:55 Consumer API 16:17:55 ✓ returns undefined if no jobs are available (7 ms) 16:17:55 ✓ returns jobs in priority order (1 ms) 16:17:55 ✓ returns undefined if no jobs are available for the given allowList 16:17:55 ✓ returns a job if it is in the allowList (1 ms) 16:17:55 ✓ returns the most important job if it is in the allowList (2 ms) 16:17:55 ✓ returns any job if filter is empty (2 ms) 16:17:55 ✓ returns a new job when reporting job success (2 ms) 16:17:55 ✓ returns a new job when reporting permanent error (1 ms) 16:17:55 ✓ returns a new job when reporting retry-able error (2 ms) 16:17:55 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 16:17:55 ✓ returns a new job if job is already in progress elsewhere (1058 ms) 16:17:55 ✓ avoids sending the same job to a new agent after a restart (1010 ms) 16:17:55 ✓ avoids sending a completed job to a new agent after a restart (1006 ms) 16:17:55 ✓ tracks job result if in progress (2 ms) 16:17:55 ✓ tracks job result even if job is in queue (2 ms) 16:17:55 ✓ ignores reported job error if unknown job (1 ms) 16:17:55 ✓ ignores job result if unknown job (1 ms) 16:17:55 ✓ returns undefined if no jobs are available (3 ms) 16:17:55 ✓ returns jobs in priority order (44 ms) 16:17:55 ✓ returns undefined if no jobs are available for the given allowList (20 ms) 16:17:55 ✓ returns a job if it is in the allowList (62 ms) 16:17:55 ✓ returns the most important job if it is in the allowList (58 ms) 16:17:55 ✓ returns any job if filter is empty (57 ms) 16:17:55 ✓ returns a new job when reporting job success (35 ms) 16:17:55 ✓ returns a new job when reporting permanent error (40 ms) 16:17:55 ✓ returns a new job when reporting retry-able error (27 ms) 16:17:55 ✓ returns a new job when reporting progress if current one is cancelled (35 ms) 16:17:55 ✓ returns a new job if job is already in progress elsewhere (1111 ms) 16:17:55 ✓ avoids sending the same job to a new agent after a restart (1063 ms) 16:17:55 ✓ avoids sending a completed job to a new agent after a restart (1058 ms) 16:17:55 ✓ tracks job result if in progress (55 ms) 16:17:55 ✓ tracks job result even if job is in queue (67 ms) 16:17:55 ✓ ignores reported job error if unknown job (3 ms) 16:17:55 ✓ ignores job result if unknown job (3 ms) 16:17:55 Timeouts 16:17:55 ✓ tracks in progress jobs (1 ms) 16:17:55 ✓ re-enqueues jobs that time out (108 ms) 16:17:55 ✓ cancel stale jobs that time out (237 ms) 16:17:55 ✓ keeps the jobs in progress while it is alive (203 ms) 16:17:55 ✓ tracks in progress jobs (25 ms) 16:17:55 ✓ re-enqueues jobs that time out (136 ms) 16:17:55 ✓ cancel stale jobs that time out (312 ms) 16:17:55 ✓ keeps the jobs in progress while it is alive (241 ms) 16:17:55 Retries 16:17:55 ✓ retries jobs (4 ms) 16:17:55 ✓ retries up to a maximum number of times (5 ms) 16:17:55 ✓ passing retry=false does not retry (3 ms) 16:17:55 ✓ does not retry if job is stale (30 ms) 16:17:55 ✓ retries jobs (30 ms) 16:17:55 ✓ retries up to a maximum number of times (46 ms) 16:17:55 ✓ passing retry=false does not retry (35 ms) 16:17:55 ✓ does not retry if job is stale (77 ms) 16:17:55 Database management 16:17:55 ✓ re-enqueues proof requests on start (5 ms) 16:17:55 ✓ restores proof results on start (2 ms) 16:17:55 ✓ only re-enqueues unfinished jobs (3 ms) 16:17:55 ✓ saves job when enqueued (4 ms) 16:17:55 ✓ does not retain job if database fails to save (5 ms) 16:17:55 ✓ saves job result (2 ms) 16:17:55 ✓ saves result even if database fails to save (4 ms) 16:17:55 ✓ saves job error (2 ms) 16:17:55 ✓ saves job error even if database fails to save (2 ms) 16:17:55 ✓ does not save job result if job is unknown (1 ms) 16:17:55 ✓ does not save job error if job is unknown (1 ms) 16:17:55 ✓ cleans up old jobs periodically (266 ms) 16:17:55 ✓ re-enqueues proof requests on start (69 ms) 16:17:55 ✓ restores proof results on start (89 ms) 16:17:55 ✓ only re-enqueues unfinished jobs (86 ms) 16:17:55 ✓ saves job when enqueued (32 ms) 16:17:55 ✓ does not retain job if database fails to save (7 ms) 16:17:55 ✓ saves job result (43 ms) 16:17:55 ✓ saves result even if database fails to save (36 ms) 16:17:55 ✓ saves job error (47 ms) 16:17:55 ✓ saves job error even if database fails to save (39 ms) 16:17:55 ✓ does not save job result if job is unknown (6 ms) 16:17:55 ✓ does not save job error if job is unknown (5 ms) 16:17:55 ✓ cleans up old jobs periodically (434 ms) 16:17:55 16:17:55 Test Suites: 1 passed, 1 total 16:17:55 Tests: 96 passed, 96 total 16:17:55 Snapshots: 0 total 16:17:55 Time: 13.401 s 16:17:55 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 16:17:55 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?