ACI ·

Parent Log: http://ci.aztec-labs.com/5f29047ca8c24952 Command: 2c5e865251c3e242 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/2731d9a8aebf63d05db1a786296abdf897c49490 Env: REF_NAME=gh-readonly-queue/next/pr-14988-420e0a54fb4bfe8b04d7dd479f59ed0fe3ad3b74 CURRENT_VERSION=0.87.6 CI_FULL=0 Date: Thu Jun 12 17:08:32 UTC 2025 System: ARCH=arm64 CPUS=64 MEM=247Gi HOSTNAME=pr-14988_arm64_a1-fast Resources: CPU_LIST=0-63 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 17:08:35 [17:08:35.012] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.017] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4903b43b9dc2d2a8 epochNumber=42 {"provingJobId":"1:BASE_PARITY:4903b43b9dc2d2a8"} 17:08:35 [17:08:35.020] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:502a1be7d96d30fd epochNumber=1 {"provingJobId":"1:BASE_PARITY:502a1be7d96d30fd"} 17:08:35 [17:08:35.043] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.044] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab76a234690c5108 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ab76a234690c5108"} 17:08:35 [17:08:35.045] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:51c746f7fa5bf9a2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:51c746f7fa5bf9a2"} 17:08:35 [17:08:35.047] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.047] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:777b1e73ea5df871 epochNumber=1 {"provingJobId":"1:BASE_PARITY:777b1e73ea5df871"} 17:08:35 [17:08:35.048] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:777b1e73ea5df871 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:777b1e73ea5df871"} 17:08:35 [17:08:35.050] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.051] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:48c92b35380fe2e4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:48c92b35380fe2e4"} 17:08:35 [17:08:35.052] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:48c92b35380fe2e4 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:48c92b35380fe2e4"} 17:08:35 [17:08:35.053] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:48c92b35380fe2e4 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:48c92b35380fe2e4"} 17:08:35 [17:08:35.053] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:48c92b35380fe2e4 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:48c92b35380fe2e4"} 17:08:35 [17:08:35.054] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.055] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c303a839bcbd491e epochNumber=1 {"provingJobId":"1:BASE_PARITY:c303a839bcbd491e"} 17:08:35 [17:08:35.073] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.078] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.078] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9995d16d09bcfbc3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9995d16d09bcfbc3"} 17:08:35 [17:08:35.079] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9995d16d09bcfbc3 {"provingJobId":"1:BASE_PARITY:9995d16d09bcfbc3"} 17:08:35 [17:08:35.079] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9995d16d09bcfbc3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9995d16d09bcfbc3"} 17:08:35 [17:08:35.079] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9995d16d09bcfbc3 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:9995d16d09bcfbc3"} 17:08:35 [17:08:35.081] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.081] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:babdad54d1264c4b epochNumber=1 {"provingJobId":"1:BASE_PARITY:babdad54d1264c4b"} 17:08:35 [17:08:35.082] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:babdad54d1264c4b {"provingJobId":"1:BASE_PARITY:babdad54d1264c4b"} 17:08:35 [17:08:35.082] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:babdad54d1264c4b type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:babdad54d1264c4b"} 17:08:35 [17:08:35.083] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.084] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e90a4b5466884e07 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e90a4b5466884e07"} 17:08:35 [17:08:35.085] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e90a4b5466884e07 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e90a4b5466884e07"} 17:08:35 [17:08:35.085] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e90a4b5466884e07 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e90a4b5466884e07"} 17:08:35 [17:08:35.086] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.087] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b64a7828a53bf21c epochNumber=1 {"provingJobId":"1:BASE_PARITY:b64a7828a53bf21c"} 17:08:35 [17:08:35.087] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b64a7828a53bf21c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b64a7828a53bf21c"} 17:08:35 [17:08:35.087] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b64a7828a53bf21c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b64a7828a53bf21c"} 17:08:35 [17:08:35.088] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.088] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cff1be1a11cd37c3 epochNumber=0 {"provingJobId":"1:BASE_PARITY:cff1be1a11cd37c3"} 17:08:35 [17:08:35.089] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cff1be1a11cd37c3 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:cff1be1a11cd37c3"} 17:08:35 [17:08:35.089] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:808d8337a27240eb epochNumber=0 {"provingJobId":"1:BASE_PARITY:808d8337a27240eb"} 17:08:35 [17:08:35.089] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cff1be1a11cd37c3 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:cff1be1a11cd37c3"} 17:08:35 [17:08:35.089] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8bc200517803eb1f epochNumber=0 {"provingJobId":"1:BASE_PARITY:8bc200517803eb1f"} 17:08:35 [17:08:35.090] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.091] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.092] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:541143f91ddf5f3b epochNumber=2 {"provingJobId":"1:BASE_PARITY:541143f91ddf5f3b"} 17:08:35 [17:08:35.092] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9c94c1b9d721caf6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9c94c1b9d721caf6"} 17:08:35 [17:08:35.093] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.094] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dfedffca2c0c2f26 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dfedffca2c0c2f26"} 17:08:35 [17:08:35.094] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.095] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9193e049c747a0a1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9193e049c747a0a1"} 17:08:35 [17:08:35.095] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3ab6794f0bec0bd9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3ab6794f0bec0bd9"} 17:08:35 [17:08:35.095] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9bd032c738ea985a epochNumber=2 {"provingJobId":"1:BASE_PARITY:9bd032c738ea985a"} 17:08:35 [17:08:35.096] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5cb1e363d7414038 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5cb1e363d7414038"} 17:08:35 [17:08:35.096] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.097] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:63c32fb75f912b6d epochNumber=1 {"provingJobId":"1:BASE_PARITY:63c32fb75f912b6d"} 17:08:35 [17:08:35.097] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:94127c625be7fa1b epochNumber=1 {"provingJobId":"1:BASE_PARITY:94127c625be7fa1b"} 17:08:35 [17:08:35.097] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:60556ddb072ffcb7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:60556ddb072ffcb7"} 17:08:35 [17:08:35.098] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c454e00dea805c10 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c454e00dea805c10"} 17:08:35 [17:08:35.098] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.099] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2e29bc247fd8ea54 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2e29bc247fd8ea54"} 17:08:35 [17:08:35.099] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:24c5d0ff978533aa epochNumber=1 {"provingJobId":"1:BASE_PARITY:24c5d0ff978533aa"} 17:08:35 [17:08:35.099] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:532c54176eb360ed epochNumber=2 {"provingJobId":"1:BASE_PARITY:532c54176eb360ed"} 17:08:35 [17:08:35.100] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3dd86f0ead9efa5f epochNumber=1 {"provingJobId":"1:BASE_PARITY:3dd86f0ead9efa5f"} 17:08:35 [17:08:35.100] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.101] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:79f3cecbf82a8525 epochNumber=1 {"provingJobId":"1:BASE_PARITY:79f3cecbf82a8525"} 17:08:35 [17:08:35.101] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:113c7ff2a25e1243 epochNumber=1 {"provingJobId":"1:BASE_PARITY:113c7ff2a25e1243"} 17:08:35 [17:08:35.102] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:79f3cecbf82a8525 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:79f3cecbf82a8525"} 17:08:35 [17:08:35.103] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.103] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8a93f573b211d05b epochNumber=1 {"provingJobId":"1:BASE_PARITY:8a93f573b211d05b"} 17:08:35 [17:08:35.103] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:48e0281118043b74 epochNumber=1 {"provingJobId":"1:BASE_PARITY:48e0281118043b74"} 17:08:35 [17:08:35.104] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8a93f573b211d05b type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:8a93f573b211d05b"} 17:08:35 [17:08:35.104] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.105] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bd42a64f678c1567 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bd42a64f678c1567"} 17:08:35 [17:08:35.105] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9d83f39a31428618 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9d83f39a31428618"} 17:08:35 [17:08:35.105] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bd42a64f678c1567 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:bd42a64f678c1567"} 17:08:35 [17:08:35.106] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.107] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:22eddf2f7828ffc9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:22eddf2f7828ffc9"} 17:08:35 [17:08:35.107] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:22eddf2f7828ffc9 {"provingJobId":"1:BASE_PARITY:22eddf2f7828ffc9"} 17:08:35 [17:08:35.107] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:22eddf2f7828ffc9 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:22eddf2f7828ffc9"} 17:08:35 [17:08:35.108] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a5213b8cd83b2c12 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a5213b8cd83b2c12"} 17:08:35 [17:08:35.108] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:22eddf2f7828ffc9 has already been completed {"provingJobId":"1:BASE_PARITY:22eddf2f7828ffc9"} 17:08:35 [17:08:35.109] INFO: prover-client:proving-broker Proving Broker started 17:08:35 [17:08:35.109] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8c46ccae7f3258a5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8c46ccae7f3258a5"} 17:08:35 [17:08:35.110] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1a9e4d896f35aa47 epochNumber=2 {"provingJobId":"1:BASE_PARITY:1a9e4d896f35aa47"} 17:08:36 [17:08:36.161] INFO: prover-client:proving-broker Proving Broker started 17:08:36 [17:08:36.162] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8c46ccae7f3258a5 settled=false {"provingJobId":"1:BASE_PARITY:8c46ccae7f3258a5","status":"pending"} 17:08:36 [17:08:36.162] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1a9e4d896f35aa47 settled=false {"provingJobId":"1:BASE_PARITY:1a9e4d896f35aa47","status":"pending"} 17:08:36 [17:08:36.163] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:8c46ccae7f3258a5 type=BASE_PARITY startedAt=1749748115110 older agent has taken job {"provingJobId":"1:BASE_PARITY:8c46ccae7f3258a5"} 17:08:36 [17:08:36.163] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8c46ccae7f3258a5 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:8c46ccae7f3258a5"} 17:08:36 [17:08:36.164] INFO: prover-client:proving-broker Proving Broker started 17:08:36 [17:08:36.164] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:abeefb3ea51f03a8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:abeefb3ea51f03a8"} 17:08:36 [17:08:36.165] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8e20a0cc52bd7b0d epochNumber=2 {"provingJobId":"1:BASE_PARITY:8e20a0cc52bd7b0d"} 17:08:37 [17:08:37.165] INFO: prover-client:proving-broker Proving Broker started 17:08:37 [17:08:37.165] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:abeefb3ea51f03a8 settled=false {"provingJobId":"1:BASE_PARITY:abeefb3ea51f03a8","status":"pending"} 17:08:37 [17:08:37.165] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8e20a0cc52bd7b0d settled=false {"provingJobId":"1:BASE_PARITY:8e20a0cc52bd7b0d","status":"pending"} 17:08:37 [17:08:37.166] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:abeefb3ea51f03a8 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:abeefb3ea51f03a8"} 17:08:37 [17:08:37.167] INFO: prover-client:proving-broker Proving Broker started 17:08:37 [17:08:37.167] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:99a8986139c89cf5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:99a8986139c89cf5"} 17:08:37 [17:08:37.167] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:99989f6437c07b14 epochNumber=2 {"provingJobId":"1:BASE_PARITY:99989f6437c07b14"} 17:08:38 [17:08:38.168] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.169] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:99a8986139c89cf5 settled=false {"provingJobId":"1:BASE_PARITY:99a8986139c89cf5","status":"pending"} 17:08:38 [17:08:38.169] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:99989f6437c07b14 settled=false {"provingJobId":"1:BASE_PARITY:99989f6437c07b14","status":"pending"} 17:08:38 [17:08:38.170] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:99a8986139c89cf5 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:99a8986139c89cf5"} 17:08:38 [17:08:38.170] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:99a8986139c89cf5 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:99a8986139c89cf5"} 17:08:38 [17:08:38.171] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.171] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e38bb6c27980cbf3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e38bb6c27980cbf3"} 17:08:38 [17:08:38.172] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:53dae7bf08eab833 epochNumber=2 {"provingJobId":"1:BASE_PARITY:53dae7bf08eab833"} 17:08:38 [17:08:38.172] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e38bb6c27980cbf3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e38bb6c27980cbf3"} 17:08:38 [17:08:38.173] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:53dae7bf08eab833 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:53dae7bf08eab833"} 17:08:38 [17:08:38.174] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.174] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03b8d8f33524783f epochNumber=1 {"provingJobId":"1:BASE_PARITY:03b8d8f33524783f"} 17:08:38 [17:08:38.174] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9bae1081f43fc29d epochNumber=2 {"provingJobId":"1:BASE_PARITY:9bae1081f43fc29d"} 17:08:38 [17:08:38.175] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:03b8d8f33524783f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:03b8d8f33524783f"} 17:08:38 [17:08:38.175] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:03b8d8f33524783f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:03b8d8f33524783f"} 17:08:38 [17:08:38.175] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9bae1081f43fc29d type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9bae1081f43fc29d"} 17:08:38 [17:08:38.176] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.177] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a3a8aa0e3a23fdd0 err=test error {"provingJoId":"1:BASE_PARITY:a3a8aa0e3a23fdd0"} 17:08:38 [17:08:38.177] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.178] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:807d7d180bc4a680 not found {"provingJobId":"1:BASE_PARITY:807d7d180bc4a680"} 17:08:38 [17:08:38.179] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.179] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:02209f067cbb07ce epochNumber=1 {"provingJobId":"1:BASE_PARITY:02209f067cbb07ce"} 17:08:38 [17:08:38.180] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.181] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5052470ae0678b73 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5052470ae0678b73"} 17:08:38 [17:08:38.282] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5052470ae0678b73 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:5052470ae0678b73"} 17:08:38 [17:08:38.288] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.288] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:40687f552b718b38 epochNumber=1 {"provingJobId":"1:BASE_PARITY:40687f552b718b38"} 17:08:38 [17:08:38.389] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:40687f552b718b38 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:40687f552b718b38"} 17:08:38 [17:08:38.395] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6740566524b099ea epochNumber=10 {"provingJobId":"1:BASE_PARITY:6740566524b099ea"} 17:08:38 [17:08:38.414] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:38 [17:08:38.521] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.522] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a9273b1d8f5fbc92 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a9273b1d8f5fbc92"} 17:08:38 [17:08:38.673] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a9273b1d8f5fbc92 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:a9273b1d8f5fbc92"} 17:08:38 [17:08:38.724] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.725] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:288dce152294d69e epochNumber=1 {"provingJobId":"1:BASE_PARITY:288dce152294d69e"} 17:08:38 [17:08:38.725] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:288dce152294d69e type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:288dce152294d69e"} 17:08:38 [17:08:38.726] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.727] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e7fabe5971826beb epochNumber=1 {"provingJobId":"1:BASE_PARITY:e7fabe5971826beb"} 17:08:38 [17:08:38.727] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e7fabe5971826beb type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:e7fabe5971826beb"} 17:08:38 [17:08:38.728] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e7fabe5971826beb type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:e7fabe5971826beb"} 17:08:38 [17:08:38.728] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.729] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:08aca0356686ac74 epochNumber=1 {"provingJobId":"1:BASE_PARITY:08aca0356686ac74"} 17:08:38 [17:08:38.729] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:08aca0356686ac74 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:08aca0356686ac74"} 17:08:38 [17:08:38.730] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.730] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2368d29afff1bf01 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2368d29afff1bf01"} 17:08:38 [17:08:38.731] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2368d29afff1bf01 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:2368d29afff1bf01"} 17:08:38 [17:08:38.731] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9704cf65ba1a74cc epochNumber=3 {"provingJobId":"1:BASE_PARITY:9704cf65ba1a74cc"} 17:08:38 [17:08:38.755] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:38 [17:08:38.756] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2368d29afff1bf01 err=test error {"provingJoId":"1:BASE_PARITY:2368d29afff1bf01"} 17:08:38 [17:08:38.758] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.758] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1cc79e5495ff6777 settled=false {"provingJobId":"1:BASE_PARITY:1cc79e5495ff6777","status":"pending"} 17:08:38 [17:08:38.758] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c6acc067a13e79e0 settled=false {"provingJobId":"1:BASE_PARITY:c6acc067a13e79e0","status":"pending"} 17:08:38 [17:08:38.760] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.760] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3f4ca02b7cdf378e settled=true {"provingJobId":"1:BASE_PARITY:3f4ca02b7cdf378e","status":"fulfilled"} 17:08:38 [17:08:38.760] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:0755468cfa88e2df settled=true {"provingJobId":"2:BASE_PARITY:0755468cfa88e2df","status":"fulfilled"} 17:08:38 [17:08:38.761] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.761] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cf739f165f31046e settled=true {"provingJobId":"1:BASE_PARITY:cf739f165f31046e","status":"fulfilled"} 17:08:38 [17:08:38.762] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0a64544f330d8505 settled=false {"provingJobId":"1:BASE_PARITY:0a64544f330d8505","status":"pending"} 17:08:38 [17:08:38.763] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.764] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5bf4db07f8ce1914 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5bf4db07f8ce1914"} 17:08:38 [17:08:38.765] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.766] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0702331ee160481f epochNumber=1 {"provingJobId":"1:BASE_PARITY:0702331ee160481f"} 17:08:38 [17:08:38.766] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:0702331ee160481f: Error: db error: Error: db error 17:08:38 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:0702331ee160481f"} 17:08:38 [17:08:38.767] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.768] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e96e2d841dc83dbe epochNumber=1 {"provingJobId":"1:BASE_PARITY:e96e2d841dc83dbe"} 17:08:38 [17:08:38.768] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e96e2d841dc83dbe type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e96e2d841dc83dbe"} 17:08:38 [17:08:38.768] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e96e2d841dc83dbe type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e96e2d841dc83dbe"} 17:08:38 [17:08:38.769] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.770] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f0e87006cfb6b408 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f0e87006cfb6b408"} 17:08:38 [17:08:38.770] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f0e87006cfb6b408 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f0e87006cfb6b408"} 17:08:38 [17:08:38.770] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f0e87006cfb6b408 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f0e87006cfb6b408"} 17:08:38 [17:08:38.771] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:f0e87006cfb6b408: Error: db error 17:08:38 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:f0e87006cfb6b408"} 17:08:38 [17:08:38.772] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.772] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea67c9c5d735d01c epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea67c9c5d735d01c"} 17:08:38 [17:08:38.772] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ea67c9c5d735d01c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ea67c9c5d735d01c"} 17:08:38 [17:08:38.772] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ea67c9c5d735d01c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ea67c9c5d735d01c"} 17:08:38 [17:08:38.773] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.774] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:396e95493fe55a15 epochNumber=1 {"provingJobId":"1:BASE_PARITY:396e95493fe55a15"} 17:08:38 [17:08:38.774] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:396e95493fe55a15 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:396e95493fe55a15"} 17:08:38 [17:08:38.774] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:396e95493fe55a15 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:396e95493fe55a15"} 17:08:38 [17:08:38.775] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:396e95493fe55a15 jobErr=test error: Error: db error 17:08:38 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:396e95493fe55a15"} 17:08:38 [17:08:38.776] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.776] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2b6030eef0fc0258 not found {"provingJobId":"1:BASE_PARITY:2b6030eef0fc0258"} 17:08:38 [17:08:38.778] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.778] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ef64e6503e3753c6 err=test error {"provingJoId":"1:BASE_PARITY:ef64e6503e3753c6"} 17:08:38 [17:08:38.779] INFO: prover-client:proving-broker Proving Broker started 17:08:38 [17:08:38.790] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cb8057aec6829f58 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cb8057aec6829f58"} 17:08:38 [17:08:38.791] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cb8057aec6829f58 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cb8057aec6829f58"} 17:08:38 [17:08:38.791] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cb8057aec6829f58 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cb8057aec6829f58"} 17:08:38 [17:08:38.801] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:b63329a09ab165fe epochNumber=2 {"provingJobId":"2:BASE_PARITY:b63329a09ab165fe"} 17:08:38 [17:08:38.801] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:b63329a09ab165fe type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:b63329a09ab165fe"} 17:08:38 [17:08:38.801] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:b63329a09ab165fe type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:b63329a09ab165fe"} 17:08:38 [17:08:38.863] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:5a27c9010579c0e8 epochNumber=3 {"provingJobId":"3:BASE_PARITY:5a27c9010579c0e8"} 17:08:38 [17:08:38.880] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:38 [17:08:38.924] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:aa0bce14c0de25d0 epochNumber=4 {"provingJobId":"4:BASE_PARITY:aa0bce14c0de25d0"} 17:08:38 [17:08:38.931] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:38 [17:08:38.986] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:4b8fd2f30e57cd89 epochNumber=5 {"provingJobId":"5:BASE_PARITY:4b8fd2f30e57cd89"} 17:08:39 [17:08:39.007] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:39 [17:08:39.039] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.040] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2f0ce60ea628e5ce epochNumber=42 {"provingJobId":"1:BASE_PARITY:2f0ce60ea628e5ce"} 17:08:39 [17:08:39.041] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testKLaLlc/42 with map size 1073741824 17:08:39 [17:08:39.045] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.066] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:71d52c6e5f71468f epochNumber=1 {"provingJobId":"1:BASE_PARITY:71d52c6e5f71468f"} 17:08:39 [17:08:39.069] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.070] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:763d80b0cd255286 epochNumber=1 {"provingJobId":"1:BASE_PARITY:763d80b0cd255286"} 17:08:39 [17:08:39.071] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxymrQ4/1 with map size 1073741824 17:08:39 [17:08:39.072] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.092] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7240cb4084ef2624 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7240cb4084ef2624"} 17:08:39 [17:08:39.099] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.100] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cfb6b99874b8c868 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cfb6b99874b8c868"} 17:08:39 [17:08:39.102] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIZ84d5/1 with map size 1073741824 17:08:39 [17:08:39.103] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.120] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cfb6b99874b8c868 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:cfb6b99874b8c868"} 17:08:39 [17:08:39.123] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.123] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b5c513089afe6945 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b5c513089afe6945"} 17:08:39 [17:08:39.124] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1TOqdz/1 with map size 1073741824 17:08:39 [17:08:39.125] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.141] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b5c513089afe6945 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b5c513089afe6945"} 17:08:39 [17:08:39.142] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b5c513089afe6945 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b5c513089afe6945"} 17:08:39 [17:08:39.147] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b5c513089afe6945 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b5c513089afe6945"} 17:08:39 [17:08:39.149] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.150] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b16903758bd30392 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b16903758bd30392"} 17:08:39 [17:08:39.151] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXREIKo/1 with map size 1073741824 17:08:39 [17:08:39.152] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.169] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.171] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.171] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:052434610de23ff4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:052434610de23ff4"} 17:08:39 [17:08:39.172] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf2wY7O/1 with map size 1073741824 17:08:39 [17:08:39.173] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.187] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:052434610de23ff4 {"provingJobId":"1:BASE_PARITY:052434610de23ff4"} 17:08:39 [17:08:39.188] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:052434610de23ff4 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:052434610de23ff4"} 17:08:39 [17:08:39.188] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:052434610de23ff4 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:052434610de23ff4"} 17:08:39 [17:08:39.195] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.195] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2436a2ca61f311a0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2436a2ca61f311a0"} 17:08:39 [17:08:39.196] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbmXheH/1 with map size 1073741824 17:08:39 [17:08:39.197] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.211] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2436a2ca61f311a0 {"provingJobId":"1:BASE_PARITY:2436a2ca61f311a0"} 17:08:39 [17:08:39.212] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2436a2ca61f311a0 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:2436a2ca61f311a0"} 17:08:39 [17:08:39.219] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.219] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6de95417a9514fb5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6de95417a9514fb5"} 17:08:39 [17:08:39.220] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbtPabn/1 with map size 1073741824 17:08:39 [17:08:39.222] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.235] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6de95417a9514fb5 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:6de95417a9514fb5"} 17:08:39 [17:08:39.236] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6de95417a9514fb5 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6de95417a9514fb5"} 17:08:39 [17:08:39.242] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.243] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9dc552298d0d152f epochNumber=1 {"provingJobId":"1:BASE_PARITY:9dc552298d0d152f"} 17:08:39 [17:08:39.243] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testA2IEE3/1 with map size 1073741824 17:08:39 [17:08:39.245] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.262] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9dc552298d0d152f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9dc552298d0d152f"} 17:08:39 [17:08:39.263] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9dc552298d0d152f type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9dc552298d0d152f"} 17:08:39 [17:08:39.274] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.275] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cc37906c248aab6e epochNumber=0 {"provingJobId":"1:BASE_PARITY:cc37906c248aab6e"} 17:08:39 [17:08:39.276] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test4MGXPF/0 with map size 1073741824 17:08:39 [17:08:39.277] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.294] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cc37906c248aab6e epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:cc37906c248aab6e"} 17:08:39 [17:08:39.294] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7f1cd3ac596619e2 epochNumber=0 {"provingJobId":"1:BASE_PARITY:7f1cd3ac596619e2"} 17:08:39 [17:08:39.295] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cc37906c248aab6e epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:cc37906c248aab6e"} 17:08:39 [17:08:39.295] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c67bde80c5214e32 epochNumber=0 {"provingJobId":"1:BASE_PARITY:c67bde80c5214e32"} 17:08:39 [17:08:39.306] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.307] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.308] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cf3ba48e6a7a07d5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:cf3ba48e6a7a07d5"} 17:08:39 [17:08:39.308] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test1bCBNC/2 with map size 1073741824 17:08:39 [17:08:39.310] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.325] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:30bbc0d49f427164 epochNumber=1 {"provingJobId":"1:BASE_PARITY:30bbc0d49f427164"} 17:08:39 [17:08:39.326] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1bCBNC/1 with map size 1073741824 17:08:39 [17:08:39.327] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.343] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.344] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fff1df6ebefda5a4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:fff1df6ebefda5a4"} 17:08:39 [17:08:39.344] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8karsj/1 with map size 1073741824 17:08:39 [17:08:39.345] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.360] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.361] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae5d099ff8a0392d epochNumber=1 {"provingJobId":"1:BASE_PARITY:ae5d099ff8a0392d"} 17:08:39 [17:08:39.361] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWmikV3/1 with map size 1073741824 17:08:39 [17:08:39.362] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.377] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea1127fd5a1f23c0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea1127fd5a1f23c0"} 17:08:39 [17:08:39.382] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b126cd19ec03cdc2 epochNumber=2 {"provingJobId":"1:BASE_PARITY:b126cd19ec03cdc2"} 17:08:39 [17:08:39.383] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWmikV3/2 with map size 1073741824 17:08:39 [17:08:39.384] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.401] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:964e319d2fb596a9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:964e319d2fb596a9"} 17:08:39 [17:08:39.409] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.409] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e813f4aae767af49 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e813f4aae767af49"} 17:08:39 [17:08:39.410] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDEdPV4/1 with map size 1073741824 17:08:39 [17:08:39.412] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.426] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:971bb711ba177b52 epochNumber=1 {"provingJobId":"1:BASE_PARITY:971bb711ba177b52"} 17:08:39 [17:08:39.431] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7deb51ae8e36b118 epochNumber=2 {"provingJobId":"1:BASE_PARITY:7deb51ae8e36b118"} 17:08:39 [17:08:39.432] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDEdPV4/2 with map size 1073741824 17:08:39 [17:08:39.434] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.451] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3d98c3327cbdd46b epochNumber=1 {"provingJobId":"1:BASE_PARITY:3d98c3327cbdd46b"} 17:08:39 [17:08:39.460] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.461] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9270e0745a096594 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9270e0745a096594"} 17:08:39 [17:08:39.462] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqN59uF/1 with map size 1073741824 17:08:39 [17:08:39.463] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.478] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:626c6363dbf83ca5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:626c6363dbf83ca5"} 17:08:39 [17:08:39.482] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b850a21623b035f5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:b850a21623b035f5"} 17:08:39 [17:08:39.483] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqN59uF/2 with map size 1073741824 17:08:39 [17:08:39.484] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.499] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8a2fc2380de1a985 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8a2fc2380de1a985"} 17:08:39 [17:08:39.507] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.507] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8fe3f337d99137a7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8fe3f337d99137a7"} 17:08:39 [17:08:39.508] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testal2YX0/1 with map size 1073741824 17:08:39 [17:08:39.509] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.523] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1a06bf9ba7bdb76d epochNumber=1 {"provingJobId":"1:BASE_PARITY:1a06bf9ba7bdb76d"} 17:08:39 [17:08:39.529] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8fe3f337d99137a7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8fe3f337d99137a7"} 17:08:39 [17:08:39.538] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.539] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9f3ed966af98f81f epochNumber=1 {"provingJobId":"1:BASE_PARITY:9f3ed966af98f81f"} 17:08:39 [17:08:39.539] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaCb64T/1 with map size 1073741824 17:08:39 [17:08:39.541] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.556] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e8e7248da2c68f37 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e8e7248da2c68f37"} 17:08:39 [17:08:39.562] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9f3ed966af98f81f type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:9f3ed966af98f81f"} 17:08:39 [17:08:39.569] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.569] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c6cf89dda78398a7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c6cf89dda78398a7"} 17:08:39 [17:08:39.570] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrWr2jL/1 with map size 1073741824 17:08:39 [17:08:39.571] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.586] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3afe2193d9f7310c epochNumber=1 {"provingJobId":"1:BASE_PARITY:3afe2193d9f7310c"} 17:08:39 [17:08:39.591] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c6cf89dda78398a7 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:c6cf89dda78398a7"} 17:08:39 [17:08:39.594] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.594] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1ac6306af1070f8e epochNumber=1 {"provingJobId":"1:BASE_PARITY:1ac6306af1070f8e"} 17:08:39 [17:08:39.596] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmAXeLZ/1 with map size 1073741824 17:08:39 [17:08:39.597] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.612] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1ac6306af1070f8e {"provingJobId":"1:BASE_PARITY:1ac6306af1070f8e"} 17:08:39 [17:08:39.612] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1ac6306af1070f8e type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:1ac6306af1070f8e"} 17:08:39 [17:08:39.617] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e73cc1f4dc6a6cb3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e73cc1f4dc6a6cb3"} 17:08:39 [17:08:39.623] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1ac6306af1070f8e has already been completed {"provingJobId":"1:BASE_PARITY:1ac6306af1070f8e"} 17:08:39 [17:08:39.626] INFO: prover-client:proving-broker Proving Broker started 17:08:39 [17:08:39.627] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0592f7b45c553a64 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0592f7b45c553a64"} 17:08:39 [17:08:39.627] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testldBUoV/1 with map size 1073741824 17:08:39 [17:08:39.628] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:39 [17:08:39.642] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c1e58275c3ce0fa2 epochNumber=2 {"provingJobId":"1:BASE_PARITY:c1e58275c3ce0fa2"} 17:08:39 [17:08:39.642] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testldBUoV/2 with map size 1073741824 17:08:39 [17:08:39.644] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:40 [17:08:40.708] INFO: prover-client:proving-broker Proving Broker started 17:08:40 [17:08:40.712] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0592f7b45c553a64 settled=false {"provingJobId":"1:BASE_PARITY:0592f7b45c553a64","status":"pending"} 17:08:40 [17:08:40.713] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c1e58275c3ce0fa2 settled=false {"provingJobId":"1:BASE_PARITY:c1e58275c3ce0fa2","status":"pending"} 17:08:40 [17:08:40.714] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:0592f7b45c553a64 type=BASE_PARITY startedAt=1749748119658 older agent has taken job {"provingJobId":"1:BASE_PARITY:0592f7b45c553a64"} 17:08:40 [17:08:40.714] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0592f7b45c553a64 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:0592f7b45c553a64"} 17:08:40 [17:08:40.718] INFO: prover-client:proving-broker Proving Broker started 17:08:40 [17:08:40.718] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d2075b127da7842a epochNumber=1 {"provingJobId":"1:BASE_PARITY:d2075b127da7842a"} 17:08:40 [17:08:40.719] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testobMpz5/1 with map size 1073741824 17:08:40 [17:08:40.720] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:40 [17:08:40.737] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9adaa3870c678e01 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9adaa3870c678e01"} 17:08:40 [17:08:40.738] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testobMpz5/2 with map size 1073741824 17:08:40 [17:08:40.739] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:41 [17:08:41.754] INFO: prover-client:proving-broker Proving Broker started 17:08:41 [17:08:41.755] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d2075b127da7842a settled=false {"provingJobId":"1:BASE_PARITY:d2075b127da7842a","status":"pending"} 17:08:41 [17:08:41.757] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9adaa3870c678e01 settled=false {"provingJobId":"1:BASE_PARITY:9adaa3870c678e01","status":"pending"} 17:08:41 [17:08:41.757] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d2075b127da7842a type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:d2075b127da7842a"} 17:08:41 [17:08:41.760] INFO: prover-client:proving-broker Proving Broker started 17:08:41 [17:08:41.760] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:87e3df82f4d652c0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:87e3df82f4d652c0"} 17:08:41 [17:08:41.761] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSo3y0h/1 with map size 1073741824 17:08:41 [17:08:41.763] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:41 [17:08:41.779] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0d1191b75f01aeca epochNumber=2 {"provingJobId":"1:BASE_PARITY:0d1191b75f01aeca"} 17:08:41 [17:08:41.779] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSo3y0h/2 with map size 1073741824 17:08:41 [17:08:41.780] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:42 [17:08:42.795] INFO: prover-client:proving-broker Proving Broker started 17:08:42 [17:08:42.797] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:87e3df82f4d652c0 settled=false {"provingJobId":"1:BASE_PARITY:87e3df82f4d652c0","status":"pending"} 17:08:42 [17:08:42.798] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0d1191b75f01aeca settled=false {"provingJobId":"1:BASE_PARITY:0d1191b75f01aeca","status":"pending"} 17:08:42 [17:08:42.799] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:87e3df82f4d652c0 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:87e3df82f4d652c0"} 17:08:42 [17:08:42.799] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:87e3df82f4d652c0 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:87e3df82f4d652c0"} 17:08:42 [17:08:42.809] INFO: prover-client:proving-broker Proving Broker started 17:08:42 [17:08:42.809] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6546a611ebff9939 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6546a611ebff9939"} 17:08:42 [17:08:42.810] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLAjrZT/1 with map size 1073741824 17:08:42 [17:08:42.811] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:42 [17:08:42.826] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7cb53c553b020971 epochNumber=2 {"provingJobId":"1:BASE_PARITY:7cb53c553b020971"} 17:08:42 [17:08:42.827] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testLAjrZT/2 with map size 1073741824 17:08:42 [17:08:42.829] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:42 [17:08:42.843] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6546a611ebff9939 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6546a611ebff9939"} 17:08:42 [17:08:42.849] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7cb53c553b020971 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:7cb53c553b020971"} 17:08:42 [17:08:42.856] INFO: prover-client:proving-broker Proving Broker started 17:08:42 [17:08:42.857] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ebb90911bb6339c epochNumber=1 {"provingJobId":"1:BASE_PARITY:8ebb90911bb6339c"} 17:08:42 [17:08:42.858] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4zPst9/1 with map size 1073741824 17:08:42 [17:08:42.859] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:42 [17:08:42.873] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:05e81f326dace09d epochNumber=2 {"provingJobId":"1:BASE_PARITY:05e81f326dace09d"} 17:08:42 [17:08:42.873] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4zPst9/2 with map size 1073741824 17:08:42 [17:08:42.875] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:42 [17:08:42.888] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8ebb90911bb6339c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8ebb90911bb6339c"} 17:08:42 [17:08:42.889] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8ebb90911bb6339c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8ebb90911bb6339c"} 17:08:42 [17:08:42.894] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:05e81f326dace09d type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:05e81f326dace09d"} 17:08:42 [17:08:42.901] INFO: prover-client:proving-broker Proving Broker started 17:08:42 [17:08:42.902] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:35eee6a14130afd3 err=test error {"provingJoId":"1:BASE_PARITY:35eee6a14130afd3"} 17:08:42 [17:08:42.904] INFO: prover-client:proving-broker Proving Broker started 17:08:42 [17:08:42.905] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:fc4e8e1f1149358d not found {"provingJobId":"1:BASE_PARITY:fc4e8e1f1149358d"} 17:08:42 [17:08:42.907] INFO: prover-client:proving-broker Proving Broker started 17:08:42 [17:08:42.908] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:79224a45e9646d41 epochNumber=1 {"provingJobId":"1:BASE_PARITY:79224a45e9646d41"} 17:08:42 [17:08:42.908] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsjx1fm/1 with map size 1073741824 17:08:42 [17:08:42.909] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:42 [17:08:42.926] INFO: prover-client:proving-broker Proving Broker started 17:08:42 [17:08:42.927] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:74f865262745e851 epochNumber=1 {"provingJobId":"1:BASE_PARITY:74f865262745e851"} 17:08:42 [17:08:42.927] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIILHO4/1 with map size 1073741824 17:08:42 [17:08:42.929] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.052] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:74f865262745e851 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:74f865262745e851"} 17:08:43 [17:08:43.058] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.059] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:832cab1f57002e37 epochNumber=1 {"provingJobId":"1:BASE_PARITY:832cab1f57002e37"} 17:08:43 [17:08:43.059] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkLVuk6/1 with map size 1073741824 17:08:43 [17:08:43.060] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.184] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:832cab1f57002e37 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:832cab1f57002e37"} 17:08:43 [17:08:43.185] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cb5121e5299437a1 epochNumber=10 {"provingJobId":"1:BASE_PARITY:cb5121e5299437a1"} 17:08:43 [17:08:43.185] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testkLVuk6/10 with map size 1073741824 17:08:43 [17:08:43.186] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.209] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:43 [17:08:43.209] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 17:08:43 [17:08:43.330] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.331] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9e566fc85867040d epochNumber=1 {"provingJobId":"1:BASE_PARITY:9e566fc85867040d"} 17:08:43 [17:08:43.331] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhzR8LB/1 with map size 1073741824 17:08:43 [17:08:43.333] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.509] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9e566fc85867040d timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:9e566fc85867040d"} 17:08:43 [17:08:43.551] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.552] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b26cecab1ce8b46f epochNumber=1 {"provingJobId":"1:BASE_PARITY:b26cecab1ce8b46f"} 17:08:43 [17:08:43.553] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVK2BH7/1 with map size 1073741824 17:08:43 [17:08:43.554] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.570] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b26cecab1ce8b46f type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:b26cecab1ce8b46f"} 17:08:43 [17:08:43.572] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.573] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a6496638c0284677 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a6496638c0284677"} 17:08:43 [17:08:43.573] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6sqii4/1 with map size 1073741824 17:08:43 [17:08:43.575] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.589] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a6496638c0284677 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:a6496638c0284677"} 17:08:43 [17:08:43.590] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a6496638c0284677 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:a6496638c0284677"} 17:08:43 [17:08:43.597] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.597] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:41149d8a7f313879 epochNumber=1 {"provingJobId":"1:BASE_PARITY:41149d8a7f313879"} 17:08:43 [17:08:43.598] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZUVGjj/1 with map size 1073741824 17:08:43 [17:08:43.599] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.613] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:41149d8a7f313879 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:41149d8a7f313879"} 17:08:43 [17:08:43.620] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.621] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:709b6145fcd5bd65 epochNumber=1 {"provingJobId":"1:BASE_PARITY:709b6145fcd5bd65"} 17:08:43 [17:08:43.621] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testssjkii/1 with map size 1073741824 17:08:43 [17:08:43.623] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.636] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:709b6145fcd5bd65 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:709b6145fcd5bd65"} 17:08:43 [17:08:43.637] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9fa301120ab561c4 epochNumber=3 {"provingJobId":"1:BASE_PARITY:9fa301120ab561c4"} 17:08:43 [17:08:43.638] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testssjkii/3 with map size 1073741824 17:08:43 [17:08:43.640] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.671] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:43 [17:08:43.671] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 17:08:43 [17:08:43.679] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:709b6145fcd5bd65 err=test error {"provingJoId":"1:BASE_PARITY:709b6145fcd5bd65"} 17:08:43 [17:08:43.682] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYaP8PT/1 with map size 1073741824 17:08:43 [17:08:43.683] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.697] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYaP8PT/2 with map size 1073741824 17:08:43 [17:08:43.699] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.712] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.713] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3bcc6b06c919ee08 settled=false {"provingJobId":"1:BASE_PARITY:3bcc6b06c919ee08","status":"pending"} 17:08:43 [17:08:43.714] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:34bdafa876f26a07 settled=false {"provingJobId":"1:BASE_PARITY:34bdafa876f26a07","status":"pending"} 17:08:43 [17:08:43.718] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwhlyie/1 with map size 1073741824 17:08:43 [17:08:43.719] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.733] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwhlyie/2 with map size 1073741824 17:08:43 [17:08:43.735] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.758] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.760] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:33f3cb55ae0deba5 settled=true {"provingJobId":"1:BASE_PARITY:33f3cb55ae0deba5","status":"fulfilled"} 17:08:43 [17:08:43.762] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:bb8a8066f1e19e75 settled=true {"provingJobId":"2:BASE_PARITY:bb8a8066f1e19e75","status":"fulfilled"} 17:08:43 [17:08:43.765] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuAjTqg/1 with map size 1073741824 17:08:43 [17:08:43.767] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.788] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testuAjTqg/2 with map size 1073741824 17:08:43 [17:08:43.789] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.805] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.806] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f27ffe1bc57ee846 settled=true {"provingJobId":"1:BASE_PARITY:f27ffe1bc57ee846","status":"fulfilled"} 17:08:43 [17:08:43.808] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:46abc53bf86d1d48 settled=false {"provingJobId":"1:BASE_PARITY:46abc53bf86d1d48","status":"pending"} 17:08:43 [17:08:43.812] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.813] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1b767e791deb595c epochNumber=1 {"provingJobId":"1:BASE_PARITY:1b767e791deb595c"} 17:08:43 [17:08:43.813] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test90dFUA/1 with map size 1073741824 17:08:43 [17:08:43.814] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.833] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.833] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b8effc58a841f6b4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b8effc58a841f6b4"} 17:08:43 [17:08:43.834] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:b8effc58a841f6b4: Error: db error: Error: db error 17:08:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:b8effc58a841f6b4"} 17:08:43 [17:08:43.836] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.837] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a884c0e7c6ad0e68 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a884c0e7c6ad0e68"} 17:08:43 [17:08:43.837] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXV67SB/1 with map size 1073741824 17:08:43 [17:08:43.839] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.853] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a884c0e7c6ad0e68 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a884c0e7c6ad0e68"} 17:08:43 [17:08:43.854] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a884c0e7c6ad0e68 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:a884c0e7c6ad0e68"} 17:08:43 [17:08:43.862] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.862] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aeaaa85b607fa302 epochNumber=1 {"provingJobId":"1:BASE_PARITY:aeaaa85b607fa302"} 17:08:43 [17:08:43.863] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testC4CehB/1 with map size 1073741824 17:08:43 [17:08:43.865] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.880] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:aeaaa85b607fa302 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:aeaaa85b607fa302"} 17:08:43 [17:08:43.880] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:aeaaa85b607fa302 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:aeaaa85b607fa302"} 17:08:43 [17:08:43.881] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:aeaaa85b607fa302: Error: db error 17:08:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:aeaaa85b607fa302"} 17:08:43 [17:08:43.884] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.884] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c17e1de83757db49 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c17e1de83757db49"} 17:08:43 [17:08:43.885] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testogXn5U/1 with map size 1073741824 17:08:43 [17:08:43.886] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.901] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c17e1de83757db49 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c17e1de83757db49"} 17:08:43 [17:08:43.901] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c17e1de83757db49 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:c17e1de83757db49"} 17:08:43 [17:08:43.910] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.910] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d1790c7f6fe9b89f epochNumber=1 {"provingJobId":"1:BASE_PARITY:d1790c7f6fe9b89f"} 17:08:43 [17:08:43.911] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSBibDU/1 with map size 1073741824 17:08:43 [17:08:43.912] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.929] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d1790c7f6fe9b89f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d1790c7f6fe9b89f"} 17:08:43 [17:08:43.929] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d1790c7f6fe9b89f type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:d1790c7f6fe9b89f"} 17:08:43 [17:08:43.930] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:d1790c7f6fe9b89f jobErr=test error: Error: db error 17:08:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:d1790c7f6fe9b89f"} 17:08:43 [17:08:43.934] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.934] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:efdd7cc0097df646 not found {"provingJobId":"1:BASE_PARITY:efdd7cc0097df646"} 17:08:43 [17:08:43.937] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.938] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:342122f80d23bca0 err=test error {"provingJoId":"1:BASE_PARITY:342122f80d23bca0"} 17:08:43 [17:08:43.940] INFO: prover-client:proving-broker Proving Broker started 17:08:43 [17:08:43.951] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:540f1f22542b99f9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:540f1f22542b99f9"} 17:08:43 [17:08:43.952] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMa4jJe/1 with map size 1073741824 17:08:43 [17:08:43.954] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:43 [17:08:43.970] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:540f1f22542b99f9 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:540f1f22542b99f9"} 17:08:43 [17:08:43.971] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:540f1f22542b99f9 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:540f1f22542b99f9"} 17:08:43 [17:08:43.986] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:77f88dcffbdc4ae0 epochNumber=2 {"provingJobId":"2:BASE_PARITY:77f88dcffbdc4ae0"} 17:08:43 [17:08:43.986] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMa4jJe/2 with map size 1073741824 17:08:43 [17:08:43.988] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:44 [17:08:44.003] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:77f88dcffbdc4ae0 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:77f88dcffbdc4ae0"} 17:08:44 [17:08:44.003] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:77f88dcffbdc4ae0 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:77f88dcffbdc4ae0"} 17:08:44 [17:08:44.068] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:3a883870a5708f11 epochNumber=3 {"provingJobId":"3:BASE_PARITY:3a883870a5708f11"} 17:08:44 [17:08:44.069] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testMa4jJe/3 with map size 1073741824 17:08:44 [17:08:44.070] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:44 [17:08:44.094] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:44 [17:08:44.094] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 17:08:44 [17:08:44.147] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:6a93b09558aba73e epochNumber=4 {"provingJobId":"4:BASE_PARITY:6a93b09558aba73e"} 17:08:44 [17:08:44.147] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testMa4jJe/4 with map size 1073741824 17:08:44 [17:08:44.149] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:44 [17:08:44.172] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:44 [17:08:44.172] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 17:08:44 [17:08:44.225] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:12c8dafe74b70586 epochNumber=5 {"provingJobId":"5:BASE_PARITY:12c8dafe74b70586"} 17:08:44 [17:08:44.228] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testMa4jJe/5 with map size 1073741824 17:08:44 [17:08:44.230] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:08:44 [17:08:44.274] INFO: prover-client:proving-broker Cleaned up jobs=1 17:08:44 [17:08:44.274] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 17:08:44 PASS src/proving_broker/proving_broker.test.ts (10.949 s) 17:08:44 ProvingBroker 17:08:44 Producer API 17:08:44 ✓ refuses stale jobs (33 ms) 17:08:44 ✓ enqueues jobs (4 ms) 17:08:44 ✓ ignores duplicate jobs (4 ms) 17:08:44 ✓ reports correct status when enqueuing repeat jobs (4 ms) 17:08:44 ✓ throws an error in case of duplicate job IDs (18 ms) 17:08:44 ✓ returns not-found status for non-existing jobs (1 ms) 17:08:44 ✓ cancels jobs in queue (6 ms) 17:08:44 ✓ cancels jobs in-progress (2 ms) 17:08:44 ✓ returns job result if successful (2 ms) 17:08:44 ✓ returns job error if failed (2 ms) 17:08:44 ✓ correctly returns job status for concurrent writes (2 ms) 17:08:44 ✓ refuses stale jobs (31 ms) 17:08:44 ✓ enqueues jobs (30 ms) 17:08:44 ✓ ignores duplicate jobs (23 ms) 17:08:44 ✓ reports correct status when enqueuing repeat jobs (26 ms) 17:08:44 ✓ throws an error in case of duplicate job IDs (20 ms) 17:08:44 ✓ returns not-found status for non-existing jobs (2 ms) 17:08:44 ✓ cancels jobs in queue (24 ms) 17:08:44 ✓ cancels jobs in-progress (23 ms) 17:08:44 ✓ returns job result if successful (24 ms) 17:08:44 ✓ returns job error if failed (31 ms) 17:08:44 ✓ correctly returns job status for concurrent writes (33 ms) 17:08:44 Consumer API 17:08:44 ✓ returns undefined if no jobs are available (1 ms) 17:08:44 ✓ returns jobs in priority order (2 ms) 17:08:44 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 17:08:44 ✓ returns a job if it is in the allowList (2 ms) 17:08:44 ✓ returns the most important job if it is in the allowList (2 ms) 17:08:44 ✓ returns any job if filter is empty (2 ms) 17:08:44 ✓ returns a new job when reporting job success (2 ms) 17:08:44 ✓ returns a new job when reporting permanent error (2 ms) 17:08:44 ✓ returns a new job when reporting retry-able error (2 ms) 17:08:44 ✓ returns a new job when reporting progress if current one is cancelled (2 ms) 17:08:44 ✓ returns a new job if job is already in progress elsewhere (1054 ms) 17:08:44 ✓ avoids sending the same job to a new agent after a restart (1002 ms) 17:08:44 ✓ avoids sending a completed job to a new agent after a restart (1003 ms) 17:08:44 ✓ tracks job result if in progress (2 ms) 17:08:44 ✓ tracks job result even if job is in queue (2 ms) 17:08:44 ✓ ignores reported job error if unknown job (1 ms) 17:08:44 ✓ ignores job result if unknown job (1 ms) 17:08:44 ✓ returns undefined if no jobs are available (2 ms) 17:08:44 ✓ returns jobs in priority order (35 ms) 17:08:44 ✓ returns undefined if no jobs are available for the given allowList (16 ms) 17:08:44 ✓ returns a job if it is in the allowList (48 ms) 17:08:44 ✓ returns the most important job if it is in the allowList (52 ms) 17:08:44 ✓ returns any job if filter is empty (46 ms) 17:08:44 ✓ returns a new job when reporting job success (32 ms) 17:08:44 ✓ returns a new job when reporting permanent error (31 ms) 17:08:44 ✓ returns a new job when reporting retry-able error (24 ms) 17:08:44 ✓ returns a new job when reporting progress if current one is cancelled (32 ms) 17:08:44 ✓ returns a new job if job is already in progress elsewhere (1092 ms) 17:08:44 ✓ avoids sending the same job to a new agent after a restart (1043 ms) 17:08:44 ✓ avoids sending a completed job to a new agent after a restart (1049 ms) 17:08:44 ✓ tracks job result if in progress (47 ms) 17:08:44 ✓ tracks job result even if job is in queue (45 ms) 17:08:44 ✓ ignores reported job error if unknown job (3 ms) 17:08:44 ✓ ignores job result if unknown job (3 ms) 17:08:44 Timeouts 17:08:44 ✓ tracks in progress jobs (2 ms) 17:08:44 ✓ re-enqueues jobs that time out (107 ms) 17:08:44 ✓ cancel stale jobs that time out (234 ms) 17:08:44 ✓ keeps the jobs in progress while it is alive (202 ms) 17:08:44 ✓ tracks in progress jobs (19 ms) 17:08:44 ✓ re-enqueues jobs that time out (132 ms) 17:08:44 ✓ cancel stale jobs that time out (272 ms) 17:08:44 ✓ keeps the jobs in progress while it is alive (221 ms) 17:08:44 Retries 17:08:44 ✓ retries jobs (2 ms) 17:08:44 ✓ retries up to a maximum number of times (2 ms) 17:08:44 ✓ passing retry=false does not retry (2 ms) 17:08:44 ✓ does not retry if job is stale (27 ms) 17:08:44 ✓ retries jobs (21 ms) 17:08:44 ✓ retries up to a maximum number of times (24 ms) 17:08:44 ✓ passing retry=false does not retry (24 ms) 17:08:44 ✓ does not retry if job is stale (61 ms) 17:08:44 Database management 17:08:44 ✓ re-enqueues proof requests on start (2 ms) 17:08:44 ✓ restores proof results on start (2 ms) 17:08:44 ✓ only re-enqueues unfinished jobs (1 ms) 17:08:44 ✓ saves job when enqueued (3 ms) 17:08:44 ✓ does not retain job if database fails to save (2 ms) 17:08:44 ✓ saves job result (2 ms) 17:08:44 ✓ saves result even if database fails to save (2 ms) 17:08:44 ✓ saves job error (2 ms) 17:08:44 ✓ saves job error even if database fails to save (2 ms) 17:08:44 ✓ does not save job result if job is unknown (2 ms) 17:08:44 ✓ does not save job error if job is unknown (2 ms) 17:08:44 ✓ cleans up old jobs periodically (258 ms) 17:08:44 ✓ re-enqueues proof requests on start (36 ms) 17:08:44 ✓ restores proof results on start (46 ms) 17:08:44 ✓ only re-enqueues unfinished jobs (46 ms) 17:08:44 ✓ saves job when enqueued (21 ms) 17:08:44 ✓ does not retain job if database fails to save (4 ms) 17:08:44 ✓ saves job result (25 ms) 17:08:44 ✓ saves result even if database fails to save (22 ms) 17:08:44 ✓ saves job error (25 ms) 17:08:44 ✓ saves job error even if database fails to save (23 ms) 17:08:44 ✓ does not save job result if job is unknown (4 ms) 17:08:44 ✓ does not save job error if job is unknown (3 ms) 17:08:44 ✓ cleans up old jobs periodically (366 ms) 17:08:44 17:08:44 Test Suites: 1 passed, 1 total 17:08:44 Tests: 96 passed, 96 total 17:08:44 Snapshots: 0 total 17:08:44 Time: 11.016 s 17:08:44 Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i. 17:08:44 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?