Parent Log: http://ci.aztec-labs.com/c9d290fe0f81769d Command: 8eab3f4af2f43a8d yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/5d81047d880386c72ab2e6701b0c86ae607c3618 Env: REF_NAME=gh-readonly-queue/next/pr-15164-308037ad9b1489540aedbb57a9dac3d71282010e CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Fri Jun 20 13:09:12 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15164_amd64_x1-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 13:09:16 [13:09:16.792] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.798] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7ee8ce5979ad2117 epochNumber=42 {"provingJobId":"1:BASE_PARITY:7ee8ce5979ad2117"} 13:09:16 [13:09:16.800] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:0d3dcab1a3a7d429 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0d3dcab1a3a7d429"} 13:09:16 [13:09:16.829] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.830] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:860f4ca6f378688e epochNumber=1 {"provingJobId":"1:BASE_PARITY:860f4ca6f378688e"} 13:09:16 [13:09:16.831] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9673267058a74000 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9673267058a74000"} 13:09:16 [13:09:16.833] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.834] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:16d8ba02f7f3bf51 epochNumber=1 {"provingJobId":"1:BASE_PARITY:16d8ba02f7f3bf51"} 13:09:16 [13:09:16.835] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:16d8ba02f7f3bf51 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:16d8ba02f7f3bf51"} 13:09:16 [13:09:16.837] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.838] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5193ed49273be412 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5193ed49273be412"} 13:09:16 [13:09:16.839] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5193ed49273be412 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:5193ed49273be412"} 13:09:16 [13:09:16.839] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5193ed49273be412 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5193ed49273be412"} 13:09:16 [13:09:16.840] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5193ed49273be412 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:5193ed49273be412"} 13:09:16 [13:09:16.841] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.842] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5c0a5c368623557c epochNumber=1 {"provingJobId":"1:BASE_PARITY:5c0a5c368623557c"} 13:09:16 [13:09:16.866] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.868] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.868] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:463b95ce30e17139 epochNumber=1 {"provingJobId":"1:BASE_PARITY:463b95ce30e17139"} 13:09:16 [13:09:16.869] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:463b95ce30e17139 {"provingJobId":"1:BASE_PARITY:463b95ce30e17139"} 13:09:16 [13:09:16.869] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:463b95ce30e17139 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:463b95ce30e17139"} 13:09:16 [13:09:16.870] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:463b95ce30e17139 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:463b95ce30e17139"} 13:09:16 [13:09:16.871] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.871] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:557cd94c74d02519 epochNumber=1 {"provingJobId":"1:BASE_PARITY:557cd94c74d02519"} 13:09:16 [13:09:16.872] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:557cd94c74d02519 {"provingJobId":"1:BASE_PARITY:557cd94c74d02519"} 13:09:16 [13:09:16.872] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:557cd94c74d02519 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:557cd94c74d02519"} 13:09:16 [13:09:16.874] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.881] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6a2f0544fbe30322 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6a2f0544fbe30322"} 13:09:16 [13:09:16.882] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6a2f0544fbe30322 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:6a2f0544fbe30322"} 13:09:16 [13:09:16.882] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6a2f0544fbe30322 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6a2f0544fbe30322"} 13:09:16 [13:09:16.884] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.884] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:956a45e21030b1cb epochNumber=1 {"provingJobId":"1:BASE_PARITY:956a45e21030b1cb"} 13:09:16 [13:09:16.885] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:956a45e21030b1cb type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:956a45e21030b1cb"} 13:09:16 [13:09:16.885] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:956a45e21030b1cb type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:956a45e21030b1cb"} 13:09:16 [13:09:16.886] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.886] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:34a58f7841b5a341 epochNumber=0 {"provingJobId":"1:BASE_PARITY:34a58f7841b5a341"} 13:09:16 [13:09:16.887] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:34a58f7841b5a341 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:34a58f7841b5a341"} 13:09:16 [13:09:16.887] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d4d1c1b92f6aff4b epochNumber=0 {"provingJobId":"1:BASE_PARITY:d4d1c1b92f6aff4b"} 13:09:16 [13:09:16.887] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:34a58f7841b5a341 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:34a58f7841b5a341"} 13:09:16 [13:09:16.887] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2ffb754abf69c1a2 epochNumber=0 {"provingJobId":"1:BASE_PARITY:2ffb754abf69c1a2"} 13:09:16 [13:09:16.889] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.890] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.890] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:338577334e8ada48 epochNumber=2 {"provingJobId":"1:BASE_PARITY:338577334e8ada48"} 13:09:16 [13:09:16.890] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:69f65ed87d22cf14 epochNumber=1 {"provingJobId":"1:BASE_PARITY:69f65ed87d22cf14"} 13:09:16 [13:09:16.892] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.892] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b3dfe7b418e08660 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b3dfe7b418e08660"} 13:09:16 [13:09:16.893] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.893] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4626841fe2f9afb8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4626841fe2f9afb8"} 13:09:16 [13:09:16.894] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:227579e941fff987 epochNumber=1 {"provingJobId":"1:BASE_PARITY:227579e941fff987"} 13:09:16 [13:09:16.894] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:557f2cb9065825fc epochNumber=2 {"provingJobId":"1:BASE_PARITY:557f2cb9065825fc"} 13:09:16 [13:09:16.894] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9ff25d461eb0b28a epochNumber=1 {"provingJobId":"1:BASE_PARITY:9ff25d461eb0b28a"} 13:09:16 [13:09:16.895] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.895] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6e384884f1b8f212 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6e384884f1b8f212"} 13:09:16 [13:09:16.895] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea61d616d4b13d6e epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea61d616d4b13d6e"} 13:09:16 [13:09:16.896] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f6edef877750b479 epochNumber=2 {"provingJobId":"1:BASE_PARITY:f6edef877750b479"} 13:09:16 [13:09:16.896] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:09fe3ac868efe116 epochNumber=1 {"provingJobId":"1:BASE_PARITY:09fe3ac868efe116"} 13:09:16 [13:09:16.897] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.897] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b7e45849886eb14e epochNumber=1 {"provingJobId":"1:BASE_PARITY:b7e45849886eb14e"} 13:09:16 [13:09:16.897] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e042863543edb844 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e042863543edb844"} 13:09:16 [13:09:16.897] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:663bd9acae9a1a08 epochNumber=2 {"provingJobId":"1:BASE_PARITY:663bd9acae9a1a08"} 13:09:16 [13:09:16.898] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a4d98fee5364f56a epochNumber=1 {"provingJobId":"1:BASE_PARITY:a4d98fee5364f56a"} 13:09:16 [13:09:16.898] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.899] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bfd2f188db4dbc7c epochNumber=1 {"provingJobId":"1:BASE_PARITY:bfd2f188db4dbc7c"} 13:09:16 [13:09:16.899] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:df459fab37d596f7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:df459fab37d596f7"} 13:09:16 [13:09:16.899] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bfd2f188db4dbc7c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:bfd2f188db4dbc7c"} 13:09:16 [13:09:16.901] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.901] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:100396944cd21dfb epochNumber=1 {"provingJobId":"1:BASE_PARITY:100396944cd21dfb"} 13:09:16 [13:09:16.901] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aefb9a06a69507af epochNumber=1 {"provingJobId":"1:BASE_PARITY:aefb9a06a69507af"} 13:09:16 [13:09:16.902] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:100396944cd21dfb type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:100396944cd21dfb"} 13:09:16 [13:09:16.903] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.903] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6cdd7758eebdce49 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6cdd7758eebdce49"} 13:09:16 [13:09:16.903] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3dcbbb60fb2bf0f2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3dcbbb60fb2bf0f2"} 13:09:16 [13:09:16.904] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6cdd7758eebdce49 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:6cdd7758eebdce49"} 13:09:16 [13:09:16.905] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.905] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f644bf369a8f1d0a epochNumber=1 {"provingJobId":"1:BASE_PARITY:f644bf369a8f1d0a"} 13:09:16 [13:09:16.906] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f644bf369a8f1d0a {"provingJobId":"1:BASE_PARITY:f644bf369a8f1d0a"} 13:09:16 [13:09:16.906] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f644bf369a8f1d0a type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:f644bf369a8f1d0a"} 13:09:16 [13:09:16.906] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aedd8c86f3bb211e epochNumber=1 {"provingJobId":"1:BASE_PARITY:aedd8c86f3bb211e"} 13:09:16 [13:09:16.907] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f644bf369a8f1d0a has already been completed {"provingJobId":"1:BASE_PARITY:f644bf369a8f1d0a"} 13:09:16 [13:09:16.908] INFO: prover-client:proving-broker Proving Broker started 13:09:16 [13:09:16.908] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6af7d4656bdfbf02 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6af7d4656bdfbf02"} 13:09:16 [13:09:16.908] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b70e9cee890413b8 epochNumber=2 {"provingJobId":"1:BASE_PARITY:b70e9cee890413b8"} 13:09:17 [13:09:17.961] INFO: prover-client:proving-broker Proving Broker started 13:09:17 [13:09:17.962] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6af7d4656bdfbf02 settled=false {"provingJobId":"1:BASE_PARITY:6af7d4656bdfbf02","status":"pending"} 13:09:17 [13:09:17.962] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b70e9cee890413b8 settled=false {"provingJobId":"1:BASE_PARITY:b70e9cee890413b8","status":"pending"} 13:09:17 [13:09:17.963] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:6af7d4656bdfbf02 type=BASE_PARITY startedAt=1750424956909 older agent has taken job {"provingJobId":"1:BASE_PARITY:6af7d4656bdfbf02"} 13:09:17 [13:09:17.963] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6af7d4656bdfbf02 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:6af7d4656bdfbf02"} 13:09:17 [13:09:17.964] INFO: prover-client:proving-broker Proving Broker started 13:09:17 [13:09:17.965] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f50777a3851493a3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f50777a3851493a3"} 13:09:17 [13:09:17.965] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b816c6e62c03b4bb epochNumber=2 {"provingJobId":"1:BASE_PARITY:b816c6e62c03b4bb"} 13:09:18 [13:09:18.967] INFO: prover-client:proving-broker Proving Broker started 13:09:18 [13:09:18.968] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f50777a3851493a3 settled=false {"provingJobId":"1:BASE_PARITY:f50777a3851493a3","status":"pending"} 13:09:18 [13:09:18.968] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b816c6e62c03b4bb settled=false {"provingJobId":"1:BASE_PARITY:b816c6e62c03b4bb","status":"pending"} 13:09:18 [13:09:18.969] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f50777a3851493a3 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:f50777a3851493a3"} 13:09:18 [13:09:18.970] INFO: prover-client:proving-broker Proving Broker started 13:09:18 [13:09:18.971] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:812d5d8cc520d7f2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:812d5d8cc520d7f2"} 13:09:18 [13:09:18.971] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0526a86d18b837de epochNumber=2 {"provingJobId":"1:BASE_PARITY:0526a86d18b837de"} 13:09:19 [13:09:19.972] INFO: prover-client:proving-broker Proving Broker started 13:09:19 [13:09:19.973] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:812d5d8cc520d7f2 settled=false {"provingJobId":"1:BASE_PARITY:812d5d8cc520d7f2","status":"pending"} 13:09:19 [13:09:19.973] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0526a86d18b837de settled=false {"provingJobId":"1:BASE_PARITY:0526a86d18b837de","status":"pending"} 13:09:19 [13:09:19.974] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:812d5d8cc520d7f2 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:812d5d8cc520d7f2"} 13:09:19 [13:09:19.974] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:812d5d8cc520d7f2 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:812d5d8cc520d7f2"} 13:09:19 [13:09:19.975] INFO: prover-client:proving-broker Proving Broker started 13:09:19 [13:09:19.976] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8d4fc936c5e57074 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8d4fc936c5e57074"} 13:09:19 [13:09:19.976] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ac5ffa8a3529c056 epochNumber=2 {"provingJobId":"1:BASE_PARITY:ac5ffa8a3529c056"} 13:09:19 [13:09:19.976] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8d4fc936c5e57074 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8d4fc936c5e57074"} 13:09:19 [13:09:19.977] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ac5ffa8a3529c056 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ac5ffa8a3529c056"} 13:09:19 [13:09:19.978] INFO: prover-client:proving-broker Proving Broker started 13:09:19 [13:09:19.978] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:643d57917e12043d epochNumber=1 {"provingJobId":"1:BASE_PARITY:643d57917e12043d"} 13:09:19 [13:09:19.979] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9fd66bca4f7ee7c0 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9fd66bca4f7ee7c0"} 13:09:19 [13:09:19.979] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:643d57917e12043d type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:643d57917e12043d"} 13:09:19 [13:09:19.979] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:643d57917e12043d type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:643d57917e12043d"} 13:09:19 [13:09:19.980] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9fd66bca4f7ee7c0 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9fd66bca4f7ee7c0"} 13:09:19 [13:09:19.981] INFO: prover-client:proving-broker Proving Broker started 13:09:19 [13:09:19.981] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2212e94c7693a1ac err=test error {"provingJoId":"1:BASE_PARITY:2212e94c7693a1ac"} 13:09:19 [13:09:19.982] INFO: prover-client:proving-broker Proving Broker started 13:09:19 [13:09:19.983] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ad9a6d237285354f not found {"provingJobId":"1:BASE_PARITY:ad9a6d237285354f"} 13:09:19 [13:09:19.984] INFO: prover-client:proving-broker Proving Broker started 13:09:19 [13:09:19.985] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6dbfa7c468ed30f0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6dbfa7c468ed30f0"} 13:09:19 [13:09:19.986] INFO: prover-client:proving-broker Proving Broker started 13:09:19 [13:09:19.986] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:458a478cc482addf epochNumber=1 {"provingJobId":"1:BASE_PARITY:458a478cc482addf"} 13:09:20 [13:09:20.088] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:458a478cc482addf timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:458a478cc482addf"} 13:09:20 [13:09:20.095] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.096] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8d534ffcf6f5d08a epochNumber=1 {"provingJobId":"1:BASE_PARITY:8d534ffcf6f5d08a"} 13:09:20 [13:09:20.200] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8d534ffcf6f5d08a timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:8d534ffcf6f5d08a"} 13:09:20 [13:09:20.204] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a37a69389498db31 epochNumber=10 {"provingJobId":"1:BASE_PARITY:a37a69389498db31"} 13:09:20 [13:09:20.226] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:20 [13:09:20.331] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.332] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:deb99cb42f774065 epochNumber=1 {"provingJobId":"1:BASE_PARITY:deb99cb42f774065"} 13:09:20 [13:09:20.485] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:deb99cb42f774065 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:deb99cb42f774065"} 13:09:20 [13:09:20.537] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.538] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ecb4fd96a43ebb72 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ecb4fd96a43ebb72"} 13:09:20 [13:09:20.539] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ecb4fd96a43ebb72 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:ecb4fd96a43ebb72"} 13:09:20 [13:09:20.540] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.540] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0b633aa618fd8fc2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0b633aa618fd8fc2"} 13:09:20 [13:09:20.541] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0b633aa618fd8fc2 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:0b633aa618fd8fc2"} 13:09:20 [13:09:20.542] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0b633aa618fd8fc2 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:0b633aa618fd8fc2"} 13:09:20 [13:09:20.543] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.543] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:36b683531ec80bde epochNumber=1 {"provingJobId":"1:BASE_PARITY:36b683531ec80bde"} 13:09:20 [13:09:20.544] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:36b683531ec80bde type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:36b683531ec80bde"} 13:09:20 [13:09:20.544] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.545] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6e325b47b23d5964 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6e325b47b23d5964"} 13:09:20 [13:09:20.545] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6e325b47b23d5964 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:6e325b47b23d5964"} 13:09:20 [13:09:20.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:120b0b93f19b331b epochNumber=3 {"provingJobId":"1:BASE_PARITY:120b0b93f19b331b"} 13:09:20 [13:09:20.570] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:20 [13:09:20.571] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6e325b47b23d5964 err=test error {"provingJoId":"1:BASE_PARITY:6e325b47b23d5964"} 13:09:20 [13:09:20.573] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.573] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:83855f6dc2a79d69 settled=false {"provingJobId":"1:BASE_PARITY:83855f6dc2a79d69","status":"pending"} 13:09:20 [13:09:20.573] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a73138e18856b82d settled=false {"provingJobId":"1:BASE_PARITY:a73138e18856b82d","status":"pending"} 13:09:20 [13:09:20.575] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.575] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2709e96b5c62b779 settled=true {"provingJobId":"1:BASE_PARITY:2709e96b5c62b779","status":"fulfilled"} 13:09:20 [13:09:20.575] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:506373ba79e18d31 settled=true {"provingJobId":"2:BASE_PARITY:506373ba79e18d31","status":"fulfilled"} 13:09:20 [13:09:20.576] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.576] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:331abbcfdb689ebf settled=true {"provingJobId":"1:BASE_PARITY:331abbcfdb689ebf","status":"fulfilled"} 13:09:20 [13:09:20.577] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b36cb6f1f7b62eec settled=false {"provingJobId":"1:BASE_PARITY:b36cb6f1f7b62eec","status":"pending"} 13:09:20 [13:09:20.578] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.578] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:55e7cb44ead669aa epochNumber=1 {"provingJobId":"1:BASE_PARITY:55e7cb44ead669aa"} 13:09:20 [13:09:20.580] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.580] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0b31306b8d1298f1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0b31306b8d1298f1"} 13:09:20 [13:09:20.581] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:0b31306b8d1298f1: Error: db error: Error: db error 13:09:20 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:0b31306b8d1298f1"} 13:09:20 [13:09:20.582] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.583] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5e781d75d493c5c3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5e781d75d493c5c3"} 13:09:20 [13:09:20.583] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5e781d75d493c5c3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5e781d75d493c5c3"} 13:09:20 [13:09:20.583] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5e781d75d493c5c3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5e781d75d493c5c3"} 13:09:20 [13:09:20.584] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.585] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cd024fb809fec854 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cd024fb809fec854"} 13:09:20 [13:09:20.585] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cd024fb809fec854 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cd024fb809fec854"} 13:09:20 [13:09:20.585] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cd024fb809fec854 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cd024fb809fec854"} 13:09:20 [13:09:20.586] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:cd024fb809fec854: Error: db error 13:09:20 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:cd024fb809fec854"} 13:09:20 [13:09:20.587] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.587] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:300124d4bebdb334 epochNumber=1 {"provingJobId":"1:BASE_PARITY:300124d4bebdb334"} 13:09:20 [13:09:20.587] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:300124d4bebdb334 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:300124d4bebdb334"} 13:09:20 [13:09:20.588] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:300124d4bebdb334 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:300124d4bebdb334"} 13:09:20 [13:09:20.589] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.589] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bf5c769ae13966f6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bf5c769ae13966f6"} 13:09:20 [13:09:20.589] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bf5c769ae13966f6 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:bf5c769ae13966f6"} 13:09:20 [13:09:20.589] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bf5c769ae13966f6 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:bf5c769ae13966f6"} 13:09:20 [13:09:20.590] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:bf5c769ae13966f6 jobErr=test error: Error: db error 13:09:20 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:bf5c769ae13966f6"} 13:09:20 [13:09:20.591] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.591] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b61363223f0a74cf not found {"provingJobId":"1:BASE_PARITY:b61363223f0a74cf"} 13:09:20 [13:09:20.592] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.593] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:51add98bb6a02238 err=test error {"provingJoId":"1:BASE_PARITY:51add98bb6a02238"} 13:09:20 [13:09:20.594] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.605] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:83f6f3e9128fc58e epochNumber=1 {"provingJobId":"1:BASE_PARITY:83f6f3e9128fc58e"} 13:09:20 [13:09:20.605] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:83f6f3e9128fc58e type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:83f6f3e9128fc58e"} 13:09:20 [13:09:20.606] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:83f6f3e9128fc58e type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:83f6f3e9128fc58e"} 13:09:20 [13:09:20.616] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:bf373cf665618e06 epochNumber=2 {"provingJobId":"2:BASE_PARITY:bf373cf665618e06"} 13:09:20 [13:09:20.617] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:bf373cf665618e06 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:bf373cf665618e06"} 13:09:20 [13:09:20.617] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:bf373cf665618e06 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:bf373cf665618e06"} 13:09:20 [13:09:20.678] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:04bb2fd0800ca552 epochNumber=3 {"provingJobId":"3:BASE_PARITY:04bb2fd0800ca552"} 13:09:20 [13:09:20.696] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:20 [13:09:20.740] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:12319f2f25bfaa84 epochNumber=4 {"provingJobId":"4:BASE_PARITY:12319f2f25bfaa84"} 13:09:20 [13:09:20.746] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:20 [13:09:20.802] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:668eb0ba2a673deb epochNumber=5 {"provingJobId":"5:BASE_PARITY:668eb0ba2a673deb"} 13:09:20 [13:09:20.824] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:20 [13:09:20.856] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.857] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:79720802c68c5b4d epochNumber=42 {"provingJobId":"1:BASE_PARITY:79720802c68c5b4d"} 13:09:20 [13:09:20.858] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testDC8De8/42 with map size 1073741824 13:09:20 [13:09:20.862] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:20 [13:09:20.890] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:8cc6b4652ffe19ab epochNumber=1 {"provingJobId":"1:BASE_PARITY:8cc6b4652ffe19ab"} 13:09:20 [13:09:20.895] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.895] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0a93b3ef3bf55d07 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0a93b3ef3bf55d07"} 13:09:20 [13:09:20.896] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testapMxME/1 with map size 1073741824 13:09:20 [13:09:20.898] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:20 [13:09:20.919] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d09f6e8738571b67 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d09f6e8738571b67"} 13:09:20 [13:09:20.930] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.930] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7073ac48807e3a35 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7073ac48807e3a35"} 13:09:20 [13:09:20.931] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testD0NH6t/1 with map size 1073741824 13:09:20 [13:09:20.932] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:20 [13:09:20.949] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7073ac48807e3a35 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:7073ac48807e3a35"} 13:09:20 [13:09:20.952] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.953] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6b92917ad120c578 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6b92917ad120c578"} 13:09:20 [13:09:20.953] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test534BGr/1 with map size 1073741824 13:09:20 [13:09:20.955] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:20 [13:09:20.973] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6b92917ad120c578 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:6b92917ad120c578"} 13:09:20 [13:09:20.973] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6b92917ad120c578 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6b92917ad120c578"} 13:09:20 [13:09:20.981] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6b92917ad120c578 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:6b92917ad120c578"} 13:09:20 [13:09:20.987] INFO: prover-client:proving-broker Proving Broker started 13:09:20 [13:09:20.987] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a311279fcc2ddc2e epochNumber=1 {"provingJobId":"1:BASE_PARITY:a311279fcc2ddc2e"} 13:09:20 [13:09:20.988] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1adjok/1 with map size 1073741824 13:09:20 [13:09:20.989] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.028] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.032] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.033] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a8d586f3121eb30c epochNumber=1 {"provingJobId":"1:BASE_PARITY:a8d586f3121eb30c"} 13:09:21 [13:09:21.035] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzZEgn2/1 with map size 1073741824 13:09:21 [13:09:21.041] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.061] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a8d586f3121eb30c {"provingJobId":"1:BASE_PARITY:a8d586f3121eb30c"} 13:09:21 [13:09:21.062] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a8d586f3121eb30c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a8d586f3121eb30c"} 13:09:21 [13:09:21.063] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a8d586f3121eb30c type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:a8d586f3121eb30c"} 13:09:21 [13:09:21.075] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.076] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f5afb5defc2be702 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f5afb5defc2be702"} 13:09:21 [13:09:21.077] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testd9Lubl/1 with map size 1073741824 13:09:21 [13:09:21.078] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.100] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f5afb5defc2be702 {"provingJobId":"1:BASE_PARITY:f5afb5defc2be702"} 13:09:21 [13:09:21.101] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f5afb5defc2be702 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:f5afb5defc2be702"} 13:09:21 [13:09:21.115] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.116] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d5cf3ec8c3eae7be epochNumber=1 {"provingJobId":"1:BASE_PARITY:d5cf3ec8c3eae7be"} 13:09:21 [13:09:21.119] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAbGEo6/1 with map size 1073741824 13:09:21 [13:09:21.127] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.150] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d5cf3ec8c3eae7be type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d5cf3ec8c3eae7be"} 13:09:21 [13:09:21.151] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d5cf3ec8c3eae7be type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d5cf3ec8c3eae7be"} 13:09:21 [13:09:21.160] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.160] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ec1fd3bfcda06cbb epochNumber=1 {"provingJobId":"1:BASE_PARITY:ec1fd3bfcda06cbb"} 13:09:21 [13:09:21.161] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHPd2on/1 with map size 1073741824 13:09:21 [13:09:21.164] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.181] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ec1fd3bfcda06cbb type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ec1fd3bfcda06cbb"} 13:09:21 [13:09:21.182] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ec1fd3bfcda06cbb type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ec1fd3bfcda06cbb"} 13:09:21 [13:09:21.191] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.192] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d4b22cbe56b29d31 epochNumber=0 {"provingJobId":"1:BASE_PARITY:d4b22cbe56b29d31"} 13:09:21 [13:09:21.193] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testWv6BJY/0 with map size 1073741824 13:09:21 [13:09:21.196] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.220] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d4b22cbe56b29d31 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:d4b22cbe56b29d31"} 13:09:21 [13:09:21.221] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0df7a68f84309cbc epochNumber=0 {"provingJobId":"1:BASE_PARITY:0df7a68f84309cbc"} 13:09:21 [13:09:21.221] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d4b22cbe56b29d31 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:d4b22cbe56b29d31"} 13:09:21 [13:09:21.222] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a490602ddf5516bc epochNumber=0 {"provingJobId":"1:BASE_PARITY:a490602ddf5516bc"} 13:09:21 [13:09:21.240] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.244] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.245] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bda7bf2659248599 epochNumber=2 {"provingJobId":"1:BASE_PARITY:bda7bf2659248599"} 13:09:21 [13:09:21.246] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testiaODBt/2 with map size 1073741824 13:09:21 [13:09:21.249] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.267] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:23f7e013097e828e epochNumber=1 {"provingJobId":"1:BASE_PARITY:23f7e013097e828e"} 13:09:21 [13:09:21.268] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiaODBt/1 with map size 1073741824 13:09:21 [13:09:21.271] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.292] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.292] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e6809f99ae329ff2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e6809f99ae329ff2"} 13:09:21 [13:09:21.293] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnTHyBx/1 with map size 1073741824 13:09:21 [13:09:21.295] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.316] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.317] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:86b7469517e11854 epochNumber=1 {"provingJobId":"1:BASE_PARITY:86b7469517e11854"} 13:09:21 [13:09:21.318] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwVdfft/1 with map size 1073741824 13:09:21 [13:09:21.321] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.349] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:659b9a91868ecb79 epochNumber=1 {"provingJobId":"1:BASE_PARITY:659b9a91868ecb79"} 13:09:21 [13:09:21.361] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d33d3561c6ef7935 epochNumber=2 {"provingJobId":"1:BASE_PARITY:d33d3561c6ef7935"} 13:09:21 [13:09:21.363] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwVdfft/2 with map size 1073741824 13:09:21 [13:09:21.366] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.377] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:15a2cbf9caf85a2d epochNumber=1 {"provingJobId":"1:BASE_PARITY:15a2cbf9caf85a2d"} 13:09:21 [13:09:21.384] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.384] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b5860bc47b85c287 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b5860bc47b85c287"} 13:09:21 [13:09:21.385] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTV8fYm/1 with map size 1073741824 13:09:21 [13:09:21.388] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.397] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab020424dbf759fb epochNumber=1 {"provingJobId":"1:BASE_PARITY:ab020424dbf759fb"} 13:09:21 [13:09:21.401] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9aa795d1e3640f96 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9aa795d1e3640f96"} 13:09:21 [13:09:21.402] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTV8fYm/2 with map size 1073741824 13:09:21 [13:09:21.404] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.424] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:be8514494dc59b85 epochNumber=1 {"provingJobId":"1:BASE_PARITY:be8514494dc59b85"} 13:09:21 [13:09:21.440] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.441] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c73dda30a20b419c epochNumber=1 {"provingJobId":"1:BASE_PARITY:c73dda30a20b419c"} 13:09:21 [13:09:21.442] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1HrBjy/1 with map size 1073741824 13:09:21 [13:09:21.445] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.462] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:36bfefd694cf8905 epochNumber=1 {"provingJobId":"1:BASE_PARITY:36bfefd694cf8905"} 13:09:21 [13:09:21.468] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6dd4c42ed4ce6594 epochNumber=2 {"provingJobId":"1:BASE_PARITY:6dd4c42ed4ce6594"} 13:09:21 [13:09:21.469] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test1HrBjy/2 with map size 1073741824 13:09:21 [13:09:21.474] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.490] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f1dbaf4b58d4aaed epochNumber=1 {"provingJobId":"1:BASE_PARITY:f1dbaf4b58d4aaed"} 13:09:21 [13:09:21.498] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.498] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9e0bb42d100e6ffe epochNumber=1 {"provingJobId":"1:BASE_PARITY:9e0bb42d100e6ffe"} 13:09:21 [13:09:21.499] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBkbfEN/1 with map size 1073741824 13:09:21 [13:09:21.500] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.526] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:22717e611e550cb7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:22717e611e550cb7"} 13:09:21 [13:09:21.532] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9e0bb42d100e6ffe type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:9e0bb42d100e6ffe"} 13:09:21 [13:09:21.542] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.542] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ec73ee847579a390 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ec73ee847579a390"} 13:09:21 [13:09:21.543] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFIfDGJ/1 with map size 1073741824 13:09:21 [13:09:21.546] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.564] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e45935fcbf9720c8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e45935fcbf9720c8"} 13:09:21 [13:09:21.571] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ec73ee847579a390 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:ec73ee847579a390"} 13:09:21 [13:09:21.581] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.582] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eb6387c391b9008e epochNumber=1 {"provingJobId":"1:BASE_PARITY:eb6387c391b9008e"} 13:09:21 [13:09:21.583] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testomxoUk/1 with map size 1073741824 13:09:21 [13:09:21.584] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.600] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8b4700ec8a0ef4b0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8b4700ec8a0ef4b0"} 13:09:21 [13:09:21.606] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:eb6387c391b9008e type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:eb6387c391b9008e"} 13:09:21 [13:09:21.609] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.609] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:48f696811d440041 epochNumber=1 {"provingJobId":"1:BASE_PARITY:48f696811d440041"} 13:09:21 [13:09:21.610] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQbpT5D/1 with map size 1073741824 13:09:21 [13:09:21.611] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.637] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:48f696811d440041 {"provingJobId":"1:BASE_PARITY:48f696811d440041"} 13:09:21 [13:09:21.638] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:48f696811d440041 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:48f696811d440041"} 13:09:21 [13:09:21.644] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ec26c0a777efb624 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ec26c0a777efb624"} 13:09:21 [13:09:21.651] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:48f696811d440041 has already been completed {"provingJobId":"1:BASE_PARITY:48f696811d440041"} 13:09:21 [13:09:21.654] INFO: prover-client:proving-broker Proving Broker started 13:09:21 [13:09:21.655] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aa27e7a2779d099b epochNumber=1 {"provingJobId":"1:BASE_PARITY:aa27e7a2779d099b"} 13:09:21 [13:09:21.658] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfjeqwG/1 with map size 1073741824 13:09:21 [13:09:21.660] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:21 [13:09:21.678] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ddfb10e64f9d44a7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:ddfb10e64f9d44a7"} 13:09:21 [13:09:21.679] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfjeqwG/2 with map size 1073741824 13:09:21 [13:09:21.681] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:22 [13:09:22.751] INFO: prover-client:proving-broker Proving Broker started 13:09:22 [13:09:22.757] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:aa27e7a2779d099b settled=false {"provingJobId":"1:BASE_PARITY:aa27e7a2779d099b","status":"pending"} 13:09:22 [13:09:22.759] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ddfb10e64f9d44a7 settled=false {"provingJobId":"1:BASE_PARITY:ddfb10e64f9d44a7","status":"pending"} 13:09:22 [13:09:22.760] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:aa27e7a2779d099b type=BASE_PARITY startedAt=1750424961699 older agent has taken job {"provingJobId":"1:BASE_PARITY:aa27e7a2779d099b"} 13:09:22 [13:09:22.761] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:aa27e7a2779d099b type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:aa27e7a2779d099b"} 13:09:22 [13:09:22.765] INFO: prover-client:proving-broker Proving Broker started 13:09:22 [13:09:22.766] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a06bf3284fe4fb88 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a06bf3284fe4fb88"} 13:09:22 [13:09:22.767] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5pbvnk/1 with map size 1073741824 13:09:22 [13:09:22.771] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:22 [13:09:22.788] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7e9f81f10579b4f5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:7e9f81f10579b4f5"} 13:09:22 [13:09:22.790] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5pbvnk/2 with map size 1073741824 13:09:22 [13:09:22.795] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:23 [13:09:23.819] INFO: prover-client:proving-broker Proving Broker started 13:09:23 [13:09:23.821] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a06bf3284fe4fb88 settled=false {"provingJobId":"1:BASE_PARITY:a06bf3284fe4fb88","status":"pending"} 13:09:23 [13:09:23.823] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7e9f81f10579b4f5 settled=false {"provingJobId":"1:BASE_PARITY:7e9f81f10579b4f5","status":"pending"} 13:09:23 [13:09:23.824] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a06bf3284fe4fb88 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:a06bf3284fe4fb88"} 13:09:23 [13:09:23.829] INFO: prover-client:proving-broker Proving Broker started 13:09:23 [13:09:23.829] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:519fab4369fbe6db epochNumber=1 {"provingJobId":"1:BASE_PARITY:519fab4369fbe6db"} 13:09:23 [13:09:23.830] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwaJlrR/1 with map size 1073741824 13:09:23 [13:09:23.833] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:23 [13:09:23.852] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:38692739c0714ac9 epochNumber=2 {"provingJobId":"1:BASE_PARITY:38692739c0714ac9"} 13:09:23 [13:09:23.855] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwaJlrR/2 with map size 1073741824 13:09:23 [13:09:23.860] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:24 [13:09:24.880] INFO: prover-client:proving-broker Proving Broker started 13:09:24 [13:09:24.882] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:519fab4369fbe6db settled=false {"provingJobId":"1:BASE_PARITY:519fab4369fbe6db","status":"pending"} 13:09:24 [13:09:24.884] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:38692739c0714ac9 settled=false {"provingJobId":"1:BASE_PARITY:38692739c0714ac9","status":"pending"} 13:09:24 [13:09:24.885] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:519fab4369fbe6db type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:519fab4369fbe6db"} 13:09:24 [13:09:24.885] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:519fab4369fbe6db type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:519fab4369fbe6db"} 13:09:24 [13:09:24.900] INFO: prover-client:proving-broker Proving Broker started 13:09:24 [13:09:24.901] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:72d16471c2a0313a epochNumber=1 {"provingJobId":"1:BASE_PARITY:72d16471c2a0313a"} 13:09:24 [13:09:24.902] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcxovjM/1 with map size 1073741824 13:09:24 [13:09:24.904] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:24 [13:09:24.928] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1a490f5b9087c422 epochNumber=2 {"provingJobId":"1:BASE_PARITY:1a490f5b9087c422"} 13:09:24 [13:09:24.933] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcxovjM/2 with map size 1073741824 13:09:24 [13:09:24.936] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:24 [13:09:24.952] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:72d16471c2a0313a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:72d16471c2a0313a"} 13:09:24 [13:09:24.960] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1a490f5b9087c422 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:1a490f5b9087c422"} 13:09:24 [13:09:24.969] INFO: prover-client:proving-broker Proving Broker started 13:09:24 [13:09:24.969] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:87c9830d4849d2e5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:87c9830d4849d2e5"} 13:09:24 [13:09:24.970] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkIK2A4/1 with map size 1073741824 13:09:24 [13:09:24.973] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:24 [13:09:24.994] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:83a87ae34f06fd23 epochNumber=2 {"provingJobId":"1:BASE_PARITY:83a87ae34f06fd23"} 13:09:24 [13:09:24.995] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkIK2A4/2 with map size 1073741824 13:09:25 [13:09:24.999] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.019] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:87c9830d4849d2e5 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:87c9830d4849d2e5"} 13:09:25 [13:09:25.020] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:87c9830d4849d2e5 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:87c9830d4849d2e5"} 13:09:25 [13:09:25.033] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:83a87ae34f06fd23 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:83a87ae34f06fd23"} 13:09:25 [13:09:25.045] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.046] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1f279b7e149ecb06 err=test error {"provingJoId":"1:BASE_PARITY:1f279b7e149ecb06"} 13:09:25 [13:09:25.050] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.051] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8f7e6fbebc863b8e not found {"provingJobId":"1:BASE_PARITY:8f7e6fbebc863b8e"} 13:09:25 [13:09:25.054] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.055] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6037bb29e8ee3e4a epochNumber=1 {"provingJobId":"1:BASE_PARITY:6037bb29e8ee3e4a"} 13:09:25 [13:09:25.056] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdrscG3/1 with map size 1073741824 13:09:25 [13:09:25.057] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.078] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.079] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3becb3362189aa65 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3becb3362189aa65"} 13:09:25 [13:09:25.079] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDDjCZs/1 with map size 1073741824 13:09:25 [13:09:25.081] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.204] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3becb3362189aa65 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:3becb3362189aa65"} 13:09:25 [13:09:25.216] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.218] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e1cb8fa9dc87d774 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e1cb8fa9dc87d774"} 13:09:25 [13:09:25.219] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTNqr9P/1 with map size 1073741824 13:09:25 [13:09:25.227] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.372] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e1cb8fa9dc87d774 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:e1cb8fa9dc87d774"} 13:09:25 [13:09:25.374] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2c7d0dd1110ed22d epochNumber=10 {"provingJobId":"1:BASE_PARITY:2c7d0dd1110ed22d"} 13:09:25 [13:09:25.375] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testTNqr9P/10 with map size 1073741824 13:09:25 [13:09:25.378] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.399] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:25 [13:09:25.399] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 13:09:25 [13:09:25.533] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.535] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:24cb7189e59a1e7c epochNumber=1 {"provingJobId":"1:BASE_PARITY:24cb7189e59a1e7c"} 13:09:25 [13:09:25.535] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwXIeB8/1 with map size 1073741824 13:09:25 [13:09:25.539] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.738] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:24cb7189e59a1e7c timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:24cb7189e59a1e7c"} 13:09:25 [13:09:25.766] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.767] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c54ec7cd42977de8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c54ec7cd42977de8"} 13:09:25 [13:09:25.768] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdLxMuy/1 with map size 1073741824 13:09:25 [13:09:25.769] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.790] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c54ec7cd42977de8 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:c54ec7cd42977de8"} 13:09:25 [13:09:25.795] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.796] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:24e8cac4f569af42 epochNumber=1 {"provingJobId":"1:BASE_PARITY:24e8cac4f569af42"} 13:09:25 [13:09:25.797] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJOAysB/1 with map size 1073741824 13:09:25 [13:09:25.799] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.826] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:24e8cac4f569af42 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:24e8cac4f569af42"} 13:09:25 [13:09:25.827] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:24e8cac4f569af42 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:24e8cac4f569af42"} 13:09:25 [13:09:25.838] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.839] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:875d72aa0cb25706 epochNumber=1 {"provingJobId":"1:BASE_PARITY:875d72aa0cb25706"} 13:09:25 [13:09:25.841] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIJbeRt/1 with map size 1073741824 13:09:25 [13:09:25.843] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.862] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:875d72aa0cb25706 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:875d72aa0cb25706"} 13:09:25 [13:09:25.872] INFO: prover-client:proving-broker Proving Broker started 13:09:25 [13:09:25.873] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:81484e8c0eac2c8c epochNumber=1 {"provingJobId":"1:BASE_PARITY:81484e8c0eac2c8c"} 13:09:25 [13:09:25.874] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvv5Zjk/1 with map size 1073741824 13:09:25 [13:09:25.875] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.892] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:81484e8c0eac2c8c type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:81484e8c0eac2c8c"} 13:09:25 [13:09:25.893] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3da9b1cf27781b7c epochNumber=3 {"provingJobId":"1:BASE_PARITY:3da9b1cf27781b7c"} 13:09:25 [13:09:25.894] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testvv5Zjk/3 with map size 1073741824 13:09:25 [13:09:25.898] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.922] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:25 [13:09:25.923] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 13:09:25 [13:09:25.943] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:81484e8c0eac2c8c err=test error {"provingJoId":"1:BASE_PARITY:81484e8c0eac2c8c"} 13:09:25 [13:09:25.951] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjRNCnI/1 with map size 1073741824 13:09:25 [13:09:25.953] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.975] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjRNCnI/2 with map size 1073741824 13:09:25 [13:09:25.978] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:25 [13:09:25.998] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.000] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:52b9b9dbfe80e8a4 settled=false {"provingJobId":"1:BASE_PARITY:52b9b9dbfe80e8a4","status":"pending"} 13:09:26 [13:09:26.002] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ac600408f0a8ff5d settled=false {"provingJobId":"1:BASE_PARITY:ac600408f0a8ff5d","status":"pending"} 13:09:26 [13:09:26.011] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoVS44z/1 with map size 1073741824 13:09:26 [13:09:26.017] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.041] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testoVS44z/2 with map size 1073741824 13:09:26 [13:09:26.044] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.075] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.077] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e9e189b736b15d5e settled=true {"provingJobId":"1:BASE_PARITY:e9e189b736b15d5e","status":"fulfilled"} 13:09:26 [13:09:26.078] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:dc3332504908397e settled=true {"provingJobId":"2:BASE_PARITY:dc3332504908397e","status":"fulfilled"} 13:09:26 [13:09:26.084] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqXhjjf/1 with map size 1073741824 13:09:26 [13:09:26.086] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.109] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqXhjjf/2 with map size 1073741824 13:09:26 [13:09:26.111] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.134] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.136] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:79deabe60fc8063b settled=true {"provingJobId":"1:BASE_PARITY:79deabe60fc8063b","status":"fulfilled"} 13:09:26 [13:09:26.139] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3656dd8ce54db501 settled=false {"provingJobId":"1:BASE_PARITY:3656dd8ce54db501","status":"pending"} 13:09:26 [13:09:26.145] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.145] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:782c17bc40a67158 epochNumber=1 {"provingJobId":"1:BASE_PARITY:782c17bc40a67158"} 13:09:26 [13:09:26.147] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAEy10G/1 with map size 1073741824 13:09:26 [13:09:26.149] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.171] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.172] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bc1d102b0c797a37 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bc1d102b0c797a37"} 13:09:26 [13:09:26.172] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:bc1d102b0c797a37: Error: db error: Error: db error 13:09:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:bc1d102b0c797a37"} 13:09:26 [13:09:26.176] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.176] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ec36506524e8b9fb epochNumber=1 {"provingJobId":"1:BASE_PARITY:ec36506524e8b9fb"} 13:09:26 [13:09:26.177] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0EHQZI/1 with map size 1073741824 13:09:26 [13:09:26.178] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.202] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ec36506524e8b9fb type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ec36506524e8b9fb"} 13:09:26 [13:09:26.203] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ec36506524e8b9fb type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ec36506524e8b9fb"} 13:09:26 [13:09:26.220] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.221] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:db0f2ff26bd462c4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:db0f2ff26bd462c4"} 13:09:26 [13:09:26.222] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testK4S45y/1 with map size 1073741824 13:09:26 [13:09:26.224] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.248] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:db0f2ff26bd462c4 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:db0f2ff26bd462c4"} 13:09:26 [13:09:26.248] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:db0f2ff26bd462c4 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:db0f2ff26bd462c4"} 13:09:26 [13:09:26.249] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:db0f2ff26bd462c4: Error: db error 13:09:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:db0f2ff26bd462c4"} 13:09:26 [13:09:26.256] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.257] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ba9486c571108973 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ba9486c571108973"} 13:09:26 [13:09:26.258] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test30KX4w/1 with map size 1073741824 13:09:26 [13:09:26.261] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.274] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ba9486c571108973 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ba9486c571108973"} 13:09:26 [13:09:26.275] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ba9486c571108973 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ba9486c571108973"} 13:09:26 [13:09:26.288] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.289] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3f899b8a76d575ea epochNumber=1 {"provingJobId":"1:BASE_PARITY:3f899b8a76d575ea"} 13:09:26 [13:09:26.290] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMCwTWs/1 with map size 1073741824 13:09:26 [13:09:26.292] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.311] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3f899b8a76d575ea type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3f899b8a76d575ea"} 13:09:26 [13:09:26.312] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3f899b8a76d575ea type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3f899b8a76d575ea"} 13:09:26 [13:09:26.313] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:3f899b8a76d575ea jobErr=test error: Error: db error 13:09:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:3f899b8a76d575ea"} 13:09:26 [13:09:26.318] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.319] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a038239d1e134d87 not found {"provingJobId":"1:BASE_PARITY:a038239d1e134d87"} 13:09:26 [13:09:26.323] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.323] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6b5b6f44cc0ebe06 err=test error {"provingJoId":"1:BASE_PARITY:6b5b6f44cc0ebe06"} 13:09:26 [13:09:26.327] INFO: prover-client:proving-broker Proving Broker started 13:09:26 [13:09:26.338] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8aa40e2efab6e3b7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8aa40e2efab6e3b7"} 13:09:26 [13:09:26.339] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testInKgIx/1 with map size 1073741824 13:09:26 [13:09:26.344] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.367] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8aa40e2efab6e3b7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8aa40e2efab6e3b7"} 13:09:26 [13:09:26.368] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8aa40e2efab6e3b7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8aa40e2efab6e3b7"} 13:09:26 [13:09:26.386] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:726d64e20443c864 epochNumber=2 {"provingJobId":"2:BASE_PARITY:726d64e20443c864"} 13:09:26 [13:09:26.387] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testInKgIx/2 with map size 1073741824 13:09:26 [13:09:26.390] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.410] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:726d64e20443c864 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:726d64e20443c864"} 13:09:26 [13:09:26.411] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:726d64e20443c864 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:726d64e20443c864"} 13:09:26 [13:09:26.482] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:b145a9b89a68b3fc epochNumber=3 {"provingJobId":"3:BASE_PARITY:b145a9b89a68b3fc"} 13:09:26 [13:09:26.484] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testInKgIx/3 with map size 1073741824 13:09:26 [13:09:26.486] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.533] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:26 [13:09:26.534] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 13:09:26 [13:09:26.571] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:725c2968f09c7731 epochNumber=4 {"provingJobId":"4:BASE_PARITY:725c2968f09c7731"} 13:09:26 [13:09:26.572] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testInKgIx/4 with map size 1073741824 13:09:26 [13:09:26.575] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.613] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:26 [13:09:26.614] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 13:09:26 [13:09:26.655] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:c1ce173b6b24e56a epochNumber=5 {"provingJobId":"5:BASE_PARITY:c1ce173b6b24e56a"} 13:09:26 [13:09:26.656] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testInKgIx/5 with map size 1073741824 13:09:26 [13:09:26.658] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 13:09:26 [13:09:26.696] INFO: prover-client:proving-broker Cleaned up jobs=1 13:09:26 [13:09:26.697] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 13:09:26 PASS src/proving_broker/proving_broker.test.ts (13.131 s) 13:09:26 ProvingBroker 13:09:26 Producer API 13:09:26 ✓ refuses stale jobs (39 ms) 13:09:26 ✓ enqueues jobs (4 ms) 13:09:26 ✓ ignores duplicate jobs (4 ms) 13:09:26 ✓ reports correct status when enqueuing repeat jobs (5 ms) 13:09:26 ✓ throws an error in case of duplicate job IDs (24 ms) 13:09:26 ✓ returns not-found status for non-existing jobs (1 ms) 13:09:26 ✓ cancels jobs in queue (3 ms) 13:09:26 ✓ cancels jobs in-progress (2 ms) 13:09:26 ✓ returns job result if successful (10 ms) 13:09:26 ✓ returns job error if failed (2 ms) 13:09:26 ✓ correctly returns job status for concurrent writes (2 ms) 13:09:26 ✓ refuses stale jobs (39 ms) 13:09:26 ✓ enqueues jobs (34 ms) 13:09:26 ✓ ignores duplicate jobs (23 ms) 13:09:26 ✓ reports correct status when enqueuing repeat jobs (32 ms) 13:09:26 ✓ throws an error in case of duplicate job IDs (34 ms) 13:09:26 ✓ returns not-found status for non-existing jobs (12 ms) 13:09:26 ✓ cancels jobs in queue (43 ms) 13:09:26 ✓ cancels jobs in-progress (36 ms) 13:09:26 ✓ returns job result if successful (48 ms) 13:09:26 ✓ returns job error if failed (31 ms) 13:09:26 ✓ correctly returns job status for concurrent writes (49 ms) 13:09:26 Consumer API 13:09:26 ✓ returns undefined if no jobs are available (1 ms) 13:09:26 ✓ returns jobs in priority order (1 ms) 13:09:26 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 13:09:26 ✓ returns a job if it is in the allowList (1 ms) 13:09:26 ✓ returns the most important job if it is in the allowList (1 ms) 13:09:26 ✓ returns any job if filter is empty (2 ms) 13:09:26 ✓ returns a new job when reporting job success (2 ms) 13:09:26 ✓ returns a new job when reporting permanent error (2 ms) 13:09:26 ✓ returns a new job when reporting retry-able error (2 ms) 13:09:26 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 13:09:26 ✓ returns a new job if job is already in progress elsewhere (1057 ms) 13:09:26 ✓ avoids sending the same job to a new agent after a restart (1005 ms) 13:09:26 ✓ avoids sending a completed job to a new agent after a restart (1005 ms) 13:09:26 ✓ tracks job result if in progress (2 ms) 13:09:26 ✓ tracks job result even if job is in queue (2 ms) 13:09:26 ✓ ignores reported job error if unknown job (2 ms) 13:09:26 ✓ ignores job result if unknown job (1 ms) 13:09:26 ✓ returns undefined if no jobs are available (2 ms) 13:09:26 ✓ returns jobs in priority order (48 ms) 13:09:26 ✓ returns undefined if no jobs are available for the given allowList (22 ms) 13:09:26 ✓ returns a job if it is in the allowList (69 ms) 13:09:26 ✓ returns the most important job if it is in the allowList (56 ms) 13:09:26 ✓ returns any job if filter is empty (58 ms) 13:09:26 ✓ returns a new job when reporting job success (43 ms) 13:09:26 ✓ returns a new job when reporting permanent error (39 ms) 13:09:26 ✓ returns a new job when reporting retry-able error (28 ms) 13:09:26 ✓ returns a new job when reporting progress if current one is cancelled (45 ms) 13:09:26 ✓ returns a new job if job is already in progress elsewhere (1110 ms) 13:09:26 ✓ avoids sending the same job to a new agent after a restart (1062 ms) 13:09:26 ✓ avoids sending a completed job to a new agent after a restart (1071 ms) 13:09:26 ✓ tracks job result if in progress (68 ms) 13:09:26 ✓ tracks job result even if job is in queue (76 ms) 13:09:26 ✓ ignores reported job error if unknown job (4 ms) 13:09:26 ✓ ignores job result if unknown job (4 ms) 13:09:26 Timeouts 13:09:26 ✓ tracks in progress jobs (2 ms) 13:09:26 ✓ re-enqueues jobs that time out (108 ms) 13:09:26 ✓ cancel stale jobs that time out (236 ms) 13:09:26 ✓ keeps the jobs in progress while it is alive (205 ms) 13:09:26 ✓ tracks in progress jobs (23 ms) 13:09:26 ✓ re-enqueues jobs that time out (135 ms) 13:09:26 ✓ cancel stale jobs that time out (318 ms) 13:09:26 ✓ keeps the jobs in progress while it is alive (234 ms) 13:09:26 Retries 13:09:26 ✓ retries jobs (2 ms) 13:09:26 ✓ retries up to a maximum number of times (3 ms) 13:09:26 ✓ passing retry=false does not retry (2 ms) 13:09:26 ✓ does not retry if job is stale (28 ms) 13:09:26 ✓ retries jobs (28 ms) 13:09:26 ✓ retries up to a maximum number of times (42 ms) 13:09:26 ✓ passing retry=false does not retry (33 ms) 13:09:26 ✓ does not retry if job is stale (76 ms) 13:09:26 Database management 13:09:26 ✓ re-enqueues proof requests on start (2 ms) 13:09:26 ✓ restores proof results on start (1 ms) 13:09:26 ✓ only re-enqueues unfinished jobs (1 ms) 13:09:26 ✓ saves job when enqueued (2 ms) 13:09:26 ✓ does not retain job if database fails to save (3 ms) 13:09:26 ✓ saves job result (2 ms) 13:09:26 ✓ saves result even if database fails to save (2 ms) 13:09:26 ✓ saves job error (2 ms) 13:09:26 ✓ saves job error even if database fails to save (2 ms) 13:09:26 ✓ does not save job result if job is unknown (1 ms) 13:09:26 ✓ does not save job error if job is unknown (1 ms) 13:09:26 ✓ cleans up old jobs periodically (261 ms) 13:09:26 ✓ re-enqueues proof requests on start (60 ms) 13:09:26 ✓ restores proof results on start (73 ms) 13:09:26 ✓ only re-enqueues unfinished jobs (61 ms) 13:09:26 ✓ saves job when enqueued (26 ms) 13:09:26 ✓ does not retain job if database fails to save (4 ms) 13:09:26 ✓ saves job result (40 ms) 13:09:26 ✓ saves result even if database fails to save (37 ms) 13:09:26 ✓ saves job error (30 ms) 13:09:26 ✓ saves job error even if database fails to save (33 ms) 13:09:26 ✓ does not save job result if job is unknown (5 ms) 13:09:26 ✓ does not save job error if job is unknown (4 ms) 13:09:26 ✓ cleans up old jobs periodically (406 ms) 13:09:26 13:09:26 Test Suites: 1 passed, 1 total 13:09:26 Tests: 96 passed, 96 total 13:09:26 Snapshots: 0 total 13:09:26 Time: 13.242 s 13:09:26 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 13:09:26 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?