Parent Log:
http://ci.aztec-labs.com/aaadd3aef9ab971c
Command: 4ef547fa5e299294 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/d4893155ff105399557b55c16450c39921c70d11
Env: REF_NAME=gh-readonly-queue/next/pr-15218-7edf16da6c056f9499e2dfd9c71fb9b7580f9bfc CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 24 10:27:31 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15218_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
10:27:35 [10:27:35.974]
INFO:
prover-client:proving-broker Proving Broker started
10:27:35 [10:27:35.982]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e54c9b54e7e095cb epochNumber=42
{"provingJobId":"1:BASE_PARITY:e54c9b54e7e095cb"}
10:27:35 [10:27:35.985]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:0b50e65bd9bedd8f epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b50e65bd9bedd8f"}
10:27:36 [10:27:36.023]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.024]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15a1258c0a6db8eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:15a1258c0a6db8eb"}
10:27:36 [10:27:36.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d086e9cafe2cff34 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d086e9cafe2cff34"}
10:27:36 [10:27:36.027]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66d66ca54693df57 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66d66ca54693df57"}
10:27:36 [10:27:36.030]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:66d66ca54693df57 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:66d66ca54693df57"}
10:27:36 [10:27:36.032]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f61590fc8c948be epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f61590fc8c948be"}
10:27:36 [10:27:36.035]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8f61590fc8c948be epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8f61590fc8c948be"}
10:27:36 [10:27:36.035]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8f61590fc8c948be type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8f61590fc8c948be"}
10:27:36 [10:27:36.036]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8f61590fc8c948be epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8f61590fc8c948be"}
10:27:36 [10:27:36.037]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.038]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b6c81f2a90c0efc epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b6c81f2a90c0efc"}
10:27:36 [10:27:36.061]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.062]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.062]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e7661ae6e4530e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e7661ae6e4530e5"}
10:27:36 [10:27:36.063]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0e7661ae6e4530e5
{"provingJobId":"1:BASE_PARITY:0e7661ae6e4530e5"}
10:27:36 [10:27:36.063]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0e7661ae6e4530e5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0e7661ae6e4530e5"}
10:27:36 [10:27:36.064]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0e7661ae6e4530e5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:0e7661ae6e4530e5"}
10:27:36 [10:27:36.065]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.066]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b29e9e15ffb8ccca epochNumber=1
{"provingJobId":"1:BASE_PARITY:b29e9e15ffb8ccca"}
10:27:36 [10:27:36.066]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b29e9e15ffb8ccca
{"provingJobId":"1:BASE_PARITY:b29e9e15ffb8ccca"}
10:27:36 [10:27:36.066]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b29e9e15ffb8ccca type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b29e9e15ffb8ccca"}
10:27:36 [10:27:36.068]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.068]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2130899197d8783 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2130899197d8783"}
10:27:36 [10:27:36.069]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f2130899197d8783 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f2130899197d8783"}
10:27:36 [10:27:36.069]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f2130899197d8783 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f2130899197d8783"}
10:27:36 [10:27:36.070]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.070]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4cb55cbf1ddc3981 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4cb55cbf1ddc3981"}
10:27:36 [10:27:36.071]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4cb55cbf1ddc3981 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4cb55cbf1ddc3981"}
10:27:36 [10:27:36.071]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4cb55cbf1ddc3981 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4cb55cbf1ddc3981"}
10:27:36 [10:27:36.072]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:535390a7fb8195cd epochNumber=0
{"provingJobId":"1:BASE_PARITY:535390a7fb8195cd"}
10:27:36 [10:27:36.073]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:535390a7fb8195cd epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:535390a7fb8195cd"}
10:27:36 [10:27:36.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b61d7cfdfb57891b epochNumber=0
{"provingJobId":"1:BASE_PARITY:b61d7cfdfb57891b"}
10:27:36 [10:27:36.074]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:535390a7fb8195cd epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:535390a7fb8195cd"}
10:27:36 [10:27:36.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6190f167be48587d epochNumber=0
{"provingJobId":"1:BASE_PARITY:6190f167be48587d"}
10:27:36 [10:27:36.081]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.082]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.083]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15f2c3f8588e96e6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:15f2c3f8588e96e6"}
10:27:36 [10:27:36.083]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7d444fb04f03fa85 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7d444fb04f03fa85"}
10:27:36 [10:27:36.084]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.084]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7617b1a3f32a03b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d7617b1a3f32a03b"}
10:27:36 [10:27:36.085]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.085]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b45c8000c539589 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b45c8000c539589"}
10:27:36 [10:27:36.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c67a91236a1d06a epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c67a91236a1d06a"}
10:27:36 [10:27:36.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df40ebac7c0fddf1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:df40ebac7c0fddf1"}
10:27:36 [10:27:36.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a1d3cb63727aba2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a1d3cb63727aba2"}
10:27:36 [10:27:36.087]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5aad41faecfcb5f epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5aad41faecfcb5f"}
10:27:36 [10:27:36.088]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86094e0c0fe94459 epochNumber=1
{"provingJobId":"1:BASE_PARITY:86094e0c0fe94459"}
10:27:36 [10:27:36.088]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae5c0957e256101e epochNumber=2
{"provingJobId":"1:BASE_PARITY:ae5c0957e256101e"}
10:27:36 [10:27:36.088]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b26395eacd6081b epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b26395eacd6081b"}
10:27:36 [10:27:36.089]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.090]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f002038a7d195a6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f002038a7d195a6c"}
10:27:36 [10:27:36.090]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:baa43f0396f249cf epochNumber=1
{"provingJobId":"1:BASE_PARITY:baa43f0396f249cf"}
10:27:36 [10:27:36.090]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd71d5abb2a2a169 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cd71d5abb2a2a169"}
10:27:36 [10:27:36.090]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4beb793380e7acfc epochNumber=1
{"provingJobId":"1:BASE_PARITY:4beb793380e7acfc"}
10:27:36 [10:27:36.091]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.092]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2187e665912b5f50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2187e665912b5f50"}
10:27:36 [10:27:36.092]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7676b8d76a95d21f epochNumber=1
{"provingJobId":"1:BASE_PARITY:7676b8d76a95d21f"}
10:27:36 [10:27:36.092]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2187e665912b5f50 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2187e665912b5f50"}
10:27:36 [10:27:36.093]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.094]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7cd7026d344d8bb9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7cd7026d344d8bb9"}
10:27:36 [10:27:36.094]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7852c8e0d73b4070 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7852c8e0d73b4070"}
10:27:36 [10:27:36.094]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7cd7026d344d8bb9 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:7cd7026d344d8bb9"}
10:27:36 [10:27:36.095]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.095]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aab3eb8d75644347 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aab3eb8d75644347"}
10:27:36 [10:27:36.096]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7cf6280faf349f45 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7cf6280faf349f45"}
10:27:36 [10:27:36.096]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:aab3eb8d75644347 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:aab3eb8d75644347"}
10:27:36 [10:27:36.097]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.097]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a8a7d6c9a4b55b48 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a8a7d6c9a4b55b48"}
10:27:36 [10:27:36.098]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a8a7d6c9a4b55b48
{"provingJobId":"1:BASE_PARITY:a8a7d6c9a4b55b48"}
10:27:36 [10:27:36.098]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a8a7d6c9a4b55b48 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a8a7d6c9a4b55b48"}
10:27:36 [10:27:36.098]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1e90da86a58227f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1e90da86a58227f"}
10:27:36 [10:27:36.099]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a8a7d6c9a4b55b48 has already been completed
{"provingJobId":"1:BASE_PARITY:a8a7d6c9a4b55b48"}
10:27:36 [10:27:36.100]
INFO:
prover-client:proving-broker Proving Broker started
10:27:36 [10:27:36.100]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bbd45517a05dca3d epochNumber=1
{"provingJobId":"1:BASE_PARITY:bbd45517a05dca3d"}
10:27:36 [10:27:36.100]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:177c8581c4624b3c epochNumber=2
{"provingJobId":"1:BASE_PARITY:177c8581c4624b3c"}
10:27:37 [10:27:37.153]
INFO:
prover-client:proving-broker Proving Broker started
10:27:37 [10:27:37.153]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bbd45517a05dca3d settled=false
{"provingJobId":"1:BASE_PARITY:bbd45517a05dca3d","status":"pending"}
10:27:37 [10:27:37.154]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:177c8581c4624b3c settled=false
{"provingJobId":"1:BASE_PARITY:177c8581c4624b3c","status":"pending"}
10:27:37 [10:27:37.155]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bbd45517a05dca3d type=BASE_PARITY startedAt=1750760856100 older agent has taken job
{"provingJobId":"1:BASE_PARITY:bbd45517a05dca3d"}
10:27:37 [10:27:37.156]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bbd45517a05dca3d type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:bbd45517a05dca3d"}
10:27:37 [10:27:37.157]
INFO:
prover-client:proving-broker Proving Broker started
10:27:37 [10:27:37.158]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:923040bd53c7e582 epochNumber=1
{"provingJobId":"1:BASE_PARITY:923040bd53c7e582"}
10:27:37 [10:27:37.158]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57f838003a988172 epochNumber=2
{"provingJobId":"1:BASE_PARITY:57f838003a988172"}
10:27:38 [10:27:38.158]
INFO:
prover-client:proving-broker Proving Broker started
10:27:38 [10:27:38.159]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:923040bd53c7e582 settled=false
{"provingJobId":"1:BASE_PARITY:923040bd53c7e582","status":"pending"}
10:27:38 [10:27:38.160]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:57f838003a988172 settled=false
{"provingJobId":"1:BASE_PARITY:57f838003a988172","status":"pending"}
10:27:38 [10:27:38.161]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:923040bd53c7e582 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:923040bd53c7e582"}
10:27:38 [10:27:38.164]
INFO:
prover-client:proving-broker Proving Broker started
10:27:38 [10:27:38.164]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7f2532769eac567 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7f2532769eac567"}
10:27:38 [10:27:38.165]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:276706b0ad0568f1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:276706b0ad0568f1"}
10:27:39 [10:27:39.168]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.168]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f7f2532769eac567 settled=false
{"provingJobId":"1:BASE_PARITY:f7f2532769eac567","status":"pending"}
10:27:39 [10:27:39.168]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:276706b0ad0568f1 settled=false
{"provingJobId":"1:BASE_PARITY:276706b0ad0568f1","status":"pending"}
10:27:39 [10:27:39.169]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f7f2532769eac567 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f7f2532769eac567"}
10:27:39 [10:27:39.169]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f7f2532769eac567 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f7f2532769eac567"}
10:27:39 [10:27:39.171]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.171]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f564613d84a99d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f564613d84a99d4"}
10:27:39 [10:27:39.171]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9fbe76c835bdedb0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9fbe76c835bdedb0"}
10:27:39 [10:27:39.172]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8f564613d84a99d4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8f564613d84a99d4"}
10:27:39 [10:27:39.173]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9fbe76c835bdedb0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9fbe76c835bdedb0"}
10:27:39 [10:27:39.174]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.174]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22f891641a524c37 epochNumber=1
{"provingJobId":"1:BASE_PARITY:22f891641a524c37"}
10:27:39 [10:27:39.174]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3276dfc3da5e1a21 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3276dfc3da5e1a21"}
10:27:39 [10:27:39.175]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:22f891641a524c37 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:22f891641a524c37"}
10:27:39 [10:27:39.175]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:22f891641a524c37 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:22f891641a524c37"}
10:27:39 [10:27:39.175]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3276dfc3da5e1a21 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3276dfc3da5e1a21"}
10:27:39 [10:27:39.177]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.177]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:67eae299ca8d9c6e err=test error
{"provingJoId":"1:BASE_PARITY:67eae299ca8d9c6e"}
10:27:39 [10:27:39.178]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.179]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:27cffefd658b306b not found
{"provingJobId":"1:BASE_PARITY:27cffefd658b306b"}
10:27:39 [10:27:39.180]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.180]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4649f2b05b72768f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4649f2b05b72768f"}
10:27:39 [10:27:39.182]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.182]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8eed7fc3957ac202 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8eed7fc3957ac202"}
10:27:39 [10:27:39.283]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8eed7fc3957ac202 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8eed7fc3957ac202"}
10:27:39 [10:27:39.290]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.290]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ac3b8889cfe0767d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ac3b8889cfe0767d"}
10:27:39 [10:27:39.393]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ac3b8889cfe0767d timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:ac3b8889cfe0767d"}
10:27:39 [10:27:39.399]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:23f59d4306861e81 epochNumber=10
{"provingJobId":"1:BASE_PARITY:23f59d4306861e81"}
10:27:39 [10:27:39.420]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:39 [10:27:39.527]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.528]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89c8cb5edabf44e4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:89c8cb5edabf44e4"}
10:27:39 [10:27:39.683]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:89c8cb5edabf44e4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:89c8cb5edabf44e4"}
10:27:39 [10:27:39.734]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.735]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eef5b29c3b3b807d epochNumber=1
{"provingJobId":"1:BASE_PARITY:eef5b29c3b3b807d"}
10:27:39 [10:27:39.736]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:eef5b29c3b3b807d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:eef5b29c3b3b807d"}
10:27:39 [10:27:39.739]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.741]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59b056bf4dc89d4e epochNumber=1
{"provingJobId":"1:BASE_PARITY:59b056bf4dc89d4e"}
10:27:39 [10:27:39.742]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:59b056bf4dc89d4e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:59b056bf4dc89d4e"}
10:27:39 [10:27:39.742]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:59b056bf4dc89d4e type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:59b056bf4dc89d4e"}
10:27:39 [10:27:39.743]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.744]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:523b131f903a6edf epochNumber=1
{"provingJobId":"1:BASE_PARITY:523b131f903a6edf"}
10:27:39 [10:27:39.744]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:523b131f903a6edf type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:523b131f903a6edf"}
10:27:39 [10:27:39.745]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.746]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:381fc78bb14b2fe2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:381fc78bb14b2fe2"}
10:27:39 [10:27:39.746]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:381fc78bb14b2fe2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:381fc78bb14b2fe2"}
10:27:39 [10:27:39.747]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0781ac725a90dbd7 epochNumber=3
{"provingJobId":"1:BASE_PARITY:0781ac725a90dbd7"}
10:27:39 [10:27:39.770]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:39 [10:27:39.773]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:381fc78bb14b2fe2 err=test error
{"provingJoId":"1:BASE_PARITY:381fc78bb14b2fe2"}
10:27:39 [10:27:39.775]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.775]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:83683e7793cb35ec settled=false
{"provingJobId":"1:BASE_PARITY:83683e7793cb35ec","status":"pending"}
10:27:39 [10:27:39.775]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:822d166e73f18b9a settled=false
{"provingJobId":"1:BASE_PARITY:822d166e73f18b9a","status":"pending"}
10:27:39 [10:27:39.777]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.777]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:072fc763d0d685bf settled=true
{"provingJobId":"1:BASE_PARITY:072fc763d0d685bf","status":"fulfilled"}
10:27:39 [10:27:39.777]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:a4ccf791b39231d4 settled=true
{"provingJobId":"2:BASE_PARITY:a4ccf791b39231d4","status":"fulfilled"}
10:27:39 [10:27:39.778]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.778]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2bf92c92aeae9c40 settled=true
{"provingJobId":"1:BASE_PARITY:2bf92c92aeae9c40","status":"fulfilled"}
10:27:39 [10:27:39.779]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b574edb5e8725be2 settled=false
{"provingJobId":"1:BASE_PARITY:b574edb5e8725be2","status":"pending"}
10:27:39 [10:27:39.780]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.780]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b295d7023404983b epochNumber=1
{"provingJobId":"1:BASE_PARITY:b295d7023404983b"}
10:27:39 [10:27:39.782]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec712c321d87e860 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec712c321d87e860"}
10:27:39 [10:27:39.783]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:ec712c321d87e860: Error: db error: Error: db error
10:27:39 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:ec712c321d87e860"}
10:27:39 [10:27:39.784]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.784]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f4395e60826b5c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f4395e60826b5c0"}
10:27:39 [10:27:39.784]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2f4395e60826b5c0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2f4395e60826b5c0"}
10:27:39 [10:27:39.784]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2f4395e60826b5c0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2f4395e60826b5c0"}
10:27:39 [10:27:39.785]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c449c460ca54a4d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c449c460ca54a4d7"}
10:27:39 [10:27:39.786]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c449c460ca54a4d7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c449c460ca54a4d7"}
10:27:39 [10:27:39.786]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c449c460ca54a4d7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c449c460ca54a4d7"}
10:27:39 [10:27:39.786]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:c449c460ca54a4d7: Error: db error
10:27:39 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:c449c460ca54a4d7"}
10:27:39 [10:27:39.787]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.787]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d81240ffbe8a4117 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d81240ffbe8a4117"}
10:27:39 [10:27:39.788]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d81240ffbe8a4117 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d81240ffbe8a4117"}
10:27:39 [10:27:39.788]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d81240ffbe8a4117 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d81240ffbe8a4117"}
10:27:39 [10:27:39.789]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.789]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a752d614df22695b epochNumber=1
{"provingJobId":"1:BASE_PARITY:a752d614df22695b"}
10:27:39 [10:27:39.789]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a752d614df22695b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a752d614df22695b"}
10:27:39 [10:27:39.789]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a752d614df22695b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a752d614df22695b"}
10:27:39 [10:27:39.790]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a752d614df22695b jobErr=test error: Error: db error
10:27:39 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:a752d614df22695b"}
10:27:39 [10:27:39.791]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.791]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:82b2f248bdcdbf70 not found
{"provingJobId":"1:BASE_PARITY:82b2f248bdcdbf70"}
10:27:39 [10:27:39.792]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.792]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a19066678bd59c79 err=test error
{"provingJoId":"1:BASE_PARITY:a19066678bd59c79"}
10:27:39 [10:27:39.793]
INFO:
prover-client:proving-broker Proving Broker started
10:27:39 [10:27:39.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:851f64e5a740e63f epochNumber=1
{"provingJobId":"1:BASE_PARITY:851f64e5a740e63f"}
10:27:39 [10:27:39.805]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:851f64e5a740e63f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:851f64e5a740e63f"}
10:27:39 [10:27:39.805]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:851f64e5a740e63f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:851f64e5a740e63f"}
10:27:39 [10:27:39.817]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:cd1806fee762fd6e epochNumber=2
{"provingJobId":"2:BASE_PARITY:cd1806fee762fd6e"}
10:27:39 [10:27:39.817]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:cd1806fee762fd6e type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:cd1806fee762fd6e"}
10:27:39 [10:27:39.818]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:cd1806fee762fd6e type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:cd1806fee762fd6e"}
10:27:39 [10:27:39.878]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:5ef664d29595f06a epochNumber=3
{"provingJobId":"3:BASE_PARITY:5ef664d29595f06a"}
10:27:39 [10:27:39.895]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:39 [10:27:39.939]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:190806c295c6550f epochNumber=4
{"provingJobId":"4:BASE_PARITY:190806c295c6550f"}
10:27:39 [10:27:39.947]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:40 [10:27:40.002]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:8644d67e8a2961f4 epochNumber=5
{"provingJobId":"5:BASE_PARITY:8644d67e8a2961f4"}
10:27:40 [10:27:40.025]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:40 [10:27:40.057]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.058]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66866d30b33b4e61 epochNumber=42
{"provingJobId":"1:BASE_PARITY:66866d30b33b4e61"}
10:27:40 [10:27:40.059]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testZENjn7/42 with map size 1073741824
10:27:40 [10:27:40.063]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.091]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:79be2a72e189e449 epochNumber=1
{"provingJobId":"1:BASE_PARITY:79be2a72e189e449"}
10:27:40 [10:27:40.095]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.096]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14bbfcf1fc8fee79 epochNumber=1
{"provingJobId":"1:BASE_PARITY:14bbfcf1fc8fee79"}
10:27:40 [10:27:40.096]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmwsSSy/1 with map size 1073741824
10:27:40 [10:27:40.098]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.118]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85b3c73bee76e145 epochNumber=1
{"provingJobId":"1:BASE_PARITY:85b3c73bee76e145"}
10:27:40 [10:27:40.127]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d447050fa28f0b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d447050fa28f0b4"}
10:27:40 [10:27:40.128]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRK5hL6/1 with map size 1073741824
10:27:40 [10:27:40.130]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.150]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8d447050fa28f0b4 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8d447050fa28f0b4"}
10:27:40 [10:27:40.154]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.155]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ebab3ba57bd562c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ebab3ba57bd562c8"}
10:27:40 [10:27:40.155]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8mWisr/1 with map size 1073741824
10:27:40 [10:27:40.158]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.176]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ebab3ba57bd562c8 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ebab3ba57bd562c8"}
10:27:40 [10:27:40.177]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ebab3ba57bd562c8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ebab3ba57bd562c8"}
10:27:40 [10:27:40.182]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ebab3ba57bd562c8 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ebab3ba57bd562c8"}
10:27:40 [10:27:40.186]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.186]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08839fbb9da000a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:08839fbb9da000a8"}
10:27:40 [10:27:40.187]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6joo1m/1 with map size 1073741824
10:27:40 [10:27:40.189]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.208]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.211]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.212]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93b23c20d90b4ae9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:93b23c20d90b4ae9"}
10:27:40 [10:27:40.213]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgt3tYF/1 with map size 1073741824
10:27:40 [10:27:40.214]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.231]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:93b23c20d90b4ae9
{"provingJobId":"1:BASE_PARITY:93b23c20d90b4ae9"}
10:27:40 [10:27:40.231]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:93b23c20d90b4ae9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:93b23c20d90b4ae9"}
10:27:40 [10:27:40.232]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:93b23c20d90b4ae9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:93b23c20d90b4ae9"}
10:27:40 [10:27:40.240]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.241]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1839fc5878586dcb epochNumber=1
{"provingJobId":"1:BASE_PARITY:1839fc5878586dcb"}
10:27:40 [10:27:40.242]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvVAYr9/1 with map size 1073741824
10:27:40 [10:27:40.243]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.260]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1839fc5878586dcb
{"provingJobId":"1:BASE_PARITY:1839fc5878586dcb"}
10:27:40 [10:27:40.260]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1839fc5878586dcb type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1839fc5878586dcb"}
10:27:40 [10:27:40.269]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.270]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:125d60f038d128c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:125d60f038d128c9"}
10:27:40 [10:27:40.270]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3W2A2A/1 with map size 1073741824
10:27:40 [10:27:40.272]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.290]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:125d60f038d128c9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:125d60f038d128c9"}
10:27:40 [10:27:40.291]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:125d60f038d128c9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:125d60f038d128c9"}
10:27:40 [10:27:40.301]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.301]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:62b175dacb8a77de epochNumber=1
{"provingJobId":"1:BASE_PARITY:62b175dacb8a77de"}
10:27:40 [10:27:40.302]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0F4GfI/1 with map size 1073741824
10:27:40 [10:27:40.305]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.329]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:62b175dacb8a77de type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:62b175dacb8a77de"}
10:27:40 [10:27:40.330]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:62b175dacb8a77de type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:62b175dacb8a77de"}
10:27:40 [10:27:40.343]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.344]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7759d08419a9fd58 epochNumber=0
{"provingJobId":"1:BASE_PARITY:7759d08419a9fd58"}
10:27:40 [10:27:40.347]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testRPsllB/0 with map size 1073741824
10:27:40 [10:27:40.350]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.368]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7759d08419a9fd58 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7759d08419a9fd58"}
10:27:40 [10:27:40.368]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:314d02955243cc9e epochNumber=0
{"provingJobId":"1:BASE_PARITY:314d02955243cc9e"}
10:27:40 [10:27:40.369]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7759d08419a9fd58 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7759d08419a9fd58"}
10:27:40 [10:27:40.369]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3421efa26c1a7b16 epochNumber=0
{"provingJobId":"1:BASE_PARITY:3421efa26c1a7b16"}
10:27:40 [10:27:40.385]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.387]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.388]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c8a4720bc330587 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7c8a4720bc330587"}
10:27:40 [10:27:40.388]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testT6bTNZ/2 with map size 1073741824
10:27:40 [10:27:40.390]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.407]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c46563104df9a4f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c46563104df9a4f3"}
10:27:40 [10:27:40.412]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testT6bTNZ/1 with map size 1073741824
10:27:40 [10:27:40.417]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.440]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.441]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:29f527aff4dd0418 epochNumber=1
{"provingJobId":"1:BASE_PARITY:29f527aff4dd0418"}
10:27:40 [10:27:40.442]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testP8ES6R/1 with map size 1073741824
10:27:40 [10:27:40.443]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.463]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.464]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28852a6b32cd6ea4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:28852a6b32cd6ea4"}
10:27:40 [10:27:40.465]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwrKrHh/1 with map size 1073741824
10:27:40 [10:27:40.467]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.485]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2703cbfe18e2235a epochNumber=1
{"provingJobId":"1:BASE_PARITY:2703cbfe18e2235a"}
10:27:40 [10:27:40.492]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ce6c7368015eae6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7ce6c7368015eae6"}
10:27:40 [10:27:40.493]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwrKrHh/2 with map size 1073741824
10:27:40 [10:27:40.496]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.523]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b7043812cac3eff epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b7043812cac3eff"}
10:27:40 [10:27:40.536]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a61e0f242ab52e6b epochNumber=1
{"provingJobId":"1:BASE_PARITY:a61e0f242ab52e6b"}
10:27:40 [10:27:40.538]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqClpQq/1 with map size 1073741824
10:27:40 [10:27:40.541]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.562]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3aa808d5efea488 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3aa808d5efea488"}
10:27:40 [10:27:40.568]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:18eeec33a6839c1e epochNumber=2
{"provingJobId":"1:BASE_PARITY:18eeec33a6839c1e"}
10:27:40 [10:27:40.569]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqClpQq/2 with map size 1073741824
10:27:40 [10:27:40.572]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.591]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7a22255390da8ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7a22255390da8ea"}
10:27:40 [10:27:40.602]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.603]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f982392f9d85eb1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f982392f9d85eb1"}
10:27:40 [10:27:40.603]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEBX4Yr/1 with map size 1073741824
10:27:40 [10:27:40.606]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.631]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d74b4b6c69073b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d74b4b6c69073b6"}
10:27:40 [10:27:40.637]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7a7311ceda67aea epochNumber=2
{"provingJobId":"1:BASE_PARITY:f7a7311ceda67aea"}
10:27:40 [10:27:40.638]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testEBX4Yr/2 with map size 1073741824
10:27:40 [10:27:40.640]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.665]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:310a2f3b96da54a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:310a2f3b96da54a4"}
10:27:40 [10:27:40.674]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.675]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:928a1c2546f48287 epochNumber=1
{"provingJobId":"1:BASE_PARITY:928a1c2546f48287"}
10:27:40 [10:27:40.676]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIegDct/1 with map size 1073741824
10:27:40 [10:27:40.678]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d8ddeac535d084b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d8ddeac535d084b"}
10:27:40 [10:27:40.700]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:928a1c2546f48287 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:928a1c2546f48287"}
10:27:40 [10:27:40.734]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.736]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e82442185aecab0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e82442185aecab0"}
10:27:40 [10:27:40.737]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTnd2VY/1 with map size 1073741824
10:27:40 [10:27:40.742]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.765]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:39c09506a8c1c06e epochNumber=1
{"provingJobId":"1:BASE_PARITY:39c09506a8c1c06e"}
10:27:40 [10:27:40.773]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3e82442185aecab0 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:3e82442185aecab0"}
10:27:40 [10:27:40.786]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f7f365c2fdcd766 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f7f365c2fdcd766"}
10:27:40 [10:27:40.789]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhZtYfg/1 with map size 1073741824
10:27:40 [10:27:40.792]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c39d9150f3deafa4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c39d9150f3deafa4"}
10:27:40 [10:27:40.826]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0f7f365c2fdcd766 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:0f7f365c2fdcd766"}
10:27:40 [10:27:40.834]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.835]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d8af28ede8181d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d8af28ede8181d1"}
10:27:40 [10:27:40.839]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-teste5AA4X/1 with map size 1073741824
10:27:40 [10:27:40.843]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.863]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6d8af28ede8181d1
{"provingJobId":"1:BASE_PARITY:6d8af28ede8181d1"}
10:27:40 [10:27:40.863]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6d8af28ede8181d1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:6d8af28ede8181d1"}
10:27:40 [10:27:40.870]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3768ee9d0acbed48 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3768ee9d0acbed48"}
10:27:40 [10:27:40.877]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6d8af28ede8181d1 has already been completed
{"provingJobId":"1:BASE_PARITY:6d8af28ede8181d1"}
10:27:40 [10:27:40.880]
INFO:
prover-client:proving-broker Proving Broker started
10:27:40 [10:27:40.881]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fc79e799c15f222 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fc79e799c15f222"}
10:27:40 [10:27:40.882]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMgBTyt/1 with map size 1073741824
10:27:40 [10:27:40.883]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:40 [10:27:40.899]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0cf87312e1044d5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e0cf87312e1044d5"}
10:27:40 [10:27:40.899]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMgBTyt/2 with map size 1073741824
10:27:40 [10:27:40.902]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:41 [10:27:41.986]
INFO:
prover-client:proving-broker Proving Broker started
10:27:41 [10:27:41.992]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0fc79e799c15f222 settled=false
{"provingJobId":"1:BASE_PARITY:0fc79e799c15f222","status":"pending"}
10:27:41 [10:27:41.994]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e0cf87312e1044d5 settled=false
{"provingJobId":"1:BASE_PARITY:e0cf87312e1044d5","status":"pending"}
10:27:41 [10:27:41.995]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0fc79e799c15f222 type=BASE_PARITY startedAt=1750760860935 older agent has taken job
{"provingJobId":"1:BASE_PARITY:0fc79e799c15f222"}
10:27:41 [10:27:41.996]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0fc79e799c15f222 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:0fc79e799c15f222"}
10:27:41 [10:27:42.000]
INFO:
prover-client:proving-broker Proving Broker started
10:27:41 [10:27:42.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b5f53d4f0d0ec38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b5f53d4f0d0ec38"}
10:27:42 [10:27:42.001]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testn8wKoJ/1 with map size 1073741824
10:27:42 [10:27:42.003]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:42 [10:27:42.026]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8cff52dc000c63d9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8cff52dc000c63d9"}
10:27:42 [10:27:42.029]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testn8wKoJ/2 with map size 1073741824
10:27:42 [10:27:42.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:43 [10:27:43.051]
INFO:
prover-client:proving-broker Proving Broker started
10:27:43 [10:27:43.052]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7b5f53d4f0d0ec38 settled=false
{"provingJobId":"1:BASE_PARITY:7b5f53d4f0d0ec38","status":"pending"}
10:27:43 [10:27:43.054]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8cff52dc000c63d9 settled=false
{"provingJobId":"1:BASE_PARITY:8cff52dc000c63d9","status":"pending"}
10:27:43 [10:27:43.055]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b5f53d4f0d0ec38 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:7b5f53d4f0d0ec38"}
10:27:43 [10:27:43.059]
INFO:
prover-client:proving-broker Proving Broker started
10:27:43 [10:27:43.060]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:333a23fe118ebd80 epochNumber=1
{"provingJobId":"1:BASE_PARITY:333a23fe118ebd80"}
10:27:43 [10:27:43.063]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testX5Ie6u/1 with map size 1073741824
10:27:43 [10:27:43.066]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:43 [10:27:43.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cafcf9b53e0fd5b4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cafcf9b53e0fd5b4"}
10:27:43 [10:27:43.088]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testX5Ie6u/2 with map size 1073741824
10:27:43 [10:27:43.092]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.118]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.120]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:333a23fe118ebd80 settled=false
{"provingJobId":"1:BASE_PARITY:333a23fe118ebd80","status":"pending"}
10:27:44 [10:27:44.122]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cafcf9b53e0fd5b4 settled=false
{"provingJobId":"1:BASE_PARITY:cafcf9b53e0fd5b4","status":"pending"}
10:27:44 [10:27:44.124]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:333a23fe118ebd80 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:333a23fe118ebd80"}
10:27:44 [10:27:44.124]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:333a23fe118ebd80 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:333a23fe118ebd80"}
10:27:44 [10:27:44.140]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.141]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a2d1d5cf6202576 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a2d1d5cf6202576"}
10:27:44 [10:27:44.142]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testS4nNQo/1 with map size 1073741824
10:27:44 [10:27:44.145]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.161]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2cfffe1ab227c867 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2cfffe1ab227c867"}
10:27:44 [10:27:44.162]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testS4nNQo/2 with map size 1073741824
10:27:44 [10:27:44.165]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.182]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2a2d1d5cf6202576 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2a2d1d5cf6202576"}
10:27:44 [10:27:44.188]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2cfffe1ab227c867 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2cfffe1ab227c867"}
10:27:44 [10:27:44.198]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.198]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:087e947960843760 epochNumber=1
{"provingJobId":"1:BASE_PARITY:087e947960843760"}
10:27:44 [10:27:44.199]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRkLErP/1 with map size 1073741824
10:27:44 [10:27:44.201]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.221]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21ff0db8f05191ce epochNumber=2
{"provingJobId":"1:BASE_PARITY:21ff0db8f05191ce"}
10:27:44 [10:27:44.222]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testRkLErP/2 with map size 1073741824
10:27:44 [10:27:44.223]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.242]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:087e947960843760 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:087e947960843760"}
10:27:44 [10:27:44.242]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:087e947960843760 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:087e947960843760"}
10:27:44 [10:27:44.249]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:21ff0db8f05191ce type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:21ff0db8f05191ce"}
10:27:44 [10:27:44.259]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.260]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a0362c814d260227 err=test error
{"provingJoId":"1:BASE_PARITY:a0362c814d260227"}
10:27:44 [10:27:44.263]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.264]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:abc4505690fb111f not found
{"provingJobId":"1:BASE_PARITY:abc4505690fb111f"}
10:27:44 [10:27:44.267]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8212e47c586a6210 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8212e47c586a6210"}
10:27:44 [10:27:44.268]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuJQCua/1 with map size 1073741824
10:27:44 [10:27:44.270]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.291]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.291]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb29b3ae4aeb03eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb29b3ae4aeb03eb"}
10:27:44 [10:27:44.292]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0VY1EX/1 with map size 1073741824
10:27:44 [10:27:44.294]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.420]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cb29b3ae4aeb03eb timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:cb29b3ae4aeb03eb"}
10:27:44 [10:27:44.431]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.432]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:137f245eb32f6745 epochNumber=1
{"provingJobId":"1:BASE_PARITY:137f245eb32f6745"}
10:27:44 [10:27:44.434]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtQzeI5/1 with map size 1073741824
10:27:44 [10:27:44.436]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.558]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:137f245eb32f6745 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:137f245eb32f6745"}
10:27:44 [10:27:44.563]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bfe50a39fb042da9 epochNumber=10
{"provingJobId":"1:BASE_PARITY:bfe50a39fb042da9"}
10:27:44 [10:27:44.564]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testtQzeI5/10 with map size 1073741824
10:27:44 [10:27:44.568]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.610]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:44 [10:27:44.611]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:27:44 [10:27:44.715]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.716]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47d7f655a33c3532 epochNumber=1
{"provingJobId":"1:BASE_PARITY:47d7f655a33c3532"}
10:27:44 [10:27:44.718]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuwvHJk/1 with map size 1073741824
10:27:44 [10:27:44.720]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.893]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:47d7f655a33c3532 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:47d7f655a33c3532"}
10:27:44 [10:27:44.942]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.943]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba522620230be8dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba522620230be8dd"}
10:27:44 [10:27:44.943]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQQUCDD/1 with map size 1073741824
10:27:44 [10:27:44.945]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.963]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ba522620230be8dd type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:ba522620230be8dd"}
10:27:44 [10:27:44.970]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:44.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06064799e788aa72 epochNumber=1
{"provingJobId":"1:BASE_PARITY:06064799e788aa72"}
10:27:44 [10:27:44.972]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhouvBG/1 with map size 1073741824
10:27:44 [10:27:44.974]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:44 [10:27:44.990]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:06064799e788aa72 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:06064799e788aa72"}
10:27:44 [10:27:44.991]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:06064799e788aa72 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:06064799e788aa72"}
10:27:44 [10:27:45.000]
INFO:
prover-client:proving-broker Proving Broker started
10:27:44 [10:27:45.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e16908e2af58ec0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e16908e2af58ec0"}
10:27:45 [10:27:45.001]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZ9nHpx/1 with map size 1073741824
10:27:45 [10:27:45.003]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.028]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4e16908e2af58ec0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4e16908e2af58ec0"}
10:27:45 [10:27:45.038]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.038]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3181581f64b3ed1a epochNumber=1
{"provingJobId":"1:BASE_PARITY:3181581f64b3ed1a"}
10:27:45 [10:27:45.039]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt3xAmM/1 with map size 1073741824
10:27:45 [10:27:45.041]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.056]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3181581f64b3ed1a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3181581f64b3ed1a"}
10:27:45 [10:27:45.058]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5189ac8ae92b2dde epochNumber=3
{"provingJobId":"1:BASE_PARITY:5189ac8ae92b2dde"}
10:27:45 [10:27:45.059]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testt3xAmM/3 with map size 1073741824
10:27:45 [10:27:45.060]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.088]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:45 [10:27:45.089]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:27:45 [10:27:45.101]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3181581f64b3ed1a err=test error
{"provingJoId":"1:BASE_PARITY:3181581f64b3ed1a"}
10:27:45 [10:27:45.107]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeKrJMN/1 with map size 1073741824
10:27:45 [10:27:45.114]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.139]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testeKrJMN/2 with map size 1073741824
10:27:45 [10:27:45.142]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.160]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.162]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d6dd33948adf359e settled=false
{"provingJobId":"1:BASE_PARITY:d6dd33948adf359e","status":"pending"}
10:27:45 [10:27:45.164]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:378c63a61f571f4c settled=false
{"provingJobId":"1:BASE_PARITY:378c63a61f571f4c","status":"pending"}
10:27:45 [10:27:45.171]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnVewav/1 with map size 1073741824
10:27:45 [10:27:45.175]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.193]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnVewav/2 with map size 1073741824
10:27:45 [10:27:45.195]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.229]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.233]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:da0c79d8329b56da settled=true
{"provingJobId":"1:BASE_PARITY:da0c79d8329b56da","status":"fulfilled"}
10:27:45 [10:27:45.234]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:43e3e7070cc59050 settled=true
{"provingJobId":"2:BASE_PARITY:43e3e7070cc59050","status":"fulfilled"}
10:27:45 [10:27:45.240]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzTl1xz/1 with map size 1073741824
10:27:45 [10:27:45.242]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.265]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testzTl1xz/2 with map size 1073741824
10:27:45 [10:27:45.267]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.283]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.285]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e2f925d27c0ad0e1 settled=true
{"provingJobId":"1:BASE_PARITY:e2f925d27c0ad0e1","status":"fulfilled"}
10:27:45 [10:27:45.287]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2316ffdcde919d18 settled=false
{"provingJobId":"1:BASE_PARITY:2316ffdcde919d18","status":"pending"}
10:27:45 [10:27:45.293]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.293]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b5aa1049905f5c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b5aa1049905f5c9"}
10:27:45 [10:27:45.294]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHzstOh/1 with map size 1073741824
10:27:45 [10:27:45.296]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.323]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.323]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c44eb2ca574f5379 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c44eb2ca574f5379"}
10:27:45 [10:27:45.324]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:c44eb2ca574f5379: Error: db error: Error: db error
10:27:45 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:c44eb2ca574f5379"}
10:27:45 [10:27:45.328]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.329]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1fea26b80df2c47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c1fea26b80df2c47"}
10:27:45 [10:27:45.331]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4NMOOR/1 with map size 1073741824
10:27:45 [10:27:45.335]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.352]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c1fea26b80df2c47 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c1fea26b80df2c47"}
10:27:45 [10:27:45.353]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c1fea26b80df2c47 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c1fea26b80df2c47"}
10:27:45 [10:27:45.365]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe370a934490a66d epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe370a934490a66d"}
10:27:45 [10:27:45.366]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvxQZRr/1 with map size 1073741824
10:27:45 [10:27:45.368]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.388]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fe370a934490a66d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fe370a934490a66d"}
10:27:45 [10:27:45.389]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fe370a934490a66d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fe370a934490a66d"}
10:27:45 [10:27:45.389]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:fe370a934490a66d: Error: db error
10:27:45 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:fe370a934490a66d"}
10:27:45 [10:27:45.395]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.395]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f21bde223b3d370d epochNumber=1
{"provingJobId":"1:BASE_PARITY:f21bde223b3d370d"}
10:27:45 [10:27:45.396]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt5Zk7F/1 with map size 1073741824
10:27:45 [10:27:45.402]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.429]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f21bde223b3d370d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f21bde223b3d370d"}
10:27:45 [10:27:45.430]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f21bde223b3d370d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f21bde223b3d370d"}
10:27:45 [10:27:45.441]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.441]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:319632090439d529 epochNumber=1
{"provingJobId":"1:BASE_PARITY:319632090439d529"}
10:27:45 [10:27:45.443]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrbP9SN/1 with map size 1073741824
10:27:45 [10:27:45.444]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.463]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:319632090439d529 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:319632090439d529"}
10:27:45 [10:27:45.464]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:319632090439d529 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:319632090439d529"}
10:27:45 [10:27:45.465]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:319632090439d529 jobErr=test error: Error: db error
10:27:45 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:319632090439d529"}
10:27:45 [10:27:45.473]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.474]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d9ab2644ac051d96 not found
{"provingJobId":"1:BASE_PARITY:d9ab2644ac051d96"}
10:27:45 [10:27:45.480]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.480]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4b8e8168c3547db3 err=test error
{"provingJoId":"1:BASE_PARITY:4b8e8168c3547db3"}
10:27:45 [10:27:45.485]
INFO:
prover-client:proving-broker Proving Broker started
10:27:45 [10:27:45.496]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e511ffb846314bc3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e511ffb846314bc3"}
10:27:45 [10:27:45.497]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKL6LTj/1 with map size 1073741824
10:27:45 [10:27:45.501]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.530]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e511ffb846314bc3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e511ffb846314bc3"}
10:27:45 [10:27:45.531]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e511ffb846314bc3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e511ffb846314bc3"}
10:27:45 [10:27:45.549]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:b0b586f0942a84b0 epochNumber=2
{"provingJobId":"2:BASE_PARITY:b0b586f0942a84b0"}
10:27:45 [10:27:45.550]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKL6LTj/2 with map size 1073741824
10:27:45 [10:27:45.554]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.573]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:b0b586f0942a84b0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:b0b586f0942a84b0"}
10:27:45 [10:27:45.574]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:b0b586f0942a84b0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:b0b586f0942a84b0"}
10:27:45 [10:27:45.642]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:c3cd7086e7351cea epochNumber=3
{"provingJobId":"3:BASE_PARITY:c3cd7086e7351cea"}
10:27:45 [10:27:45.643]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testKL6LTj/3 with map size 1073741824
10:27:45 [10:27:45.647]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.668]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:45 [10:27:45.669]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:27:45 [10:27:45.730]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:95a106e45c0b41e2 epochNumber=4
{"provingJobId":"4:BASE_PARITY:95a106e45c0b41e2"}
10:27:45 [10:27:45.732]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testKL6LTj/4 with map size 1073741824
10:27:45 [10:27:45.737]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.773]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:45 [10:27:45.774]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
10:27:45 [10:27:45.823]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:7b19cf0e1f65d584 epochNumber=5
{"provingJobId":"5:BASE_PARITY:7b19cf0e1f65d584"}
10:27:45 [10:27:45.825]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testKL6LTj/5 with map size 1073741824
10:27:45 [10:27:45.830]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:27:45 [10:27:45.878]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:27:45 [10:27:45.879]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
10:27:45
PASS src/proving_broker/proving_broker.test.ts (
13.253 s)
10:27:45 ProvingBroker
10:27:45 Producer API
10:27:45
✓ refuses stale jobs (51 ms)
10:27:45
✓ enqueues jobs (5 ms)
10:27:45
✓ ignores duplicate jobs (4 ms)
10:27:45
✓ reports correct status when enqueuing repeat jobs (4 ms)
10:27:45
✓ throws an error in case of duplicate job IDs (23 ms)
10:27:45
✓ returns not-found status for non-existing jobs (1 ms)
10:27:45
✓ cancels jobs in queue (2 ms)
10:27:45
✓ cancels jobs in-progress (2 ms)
10:27:45
✓ returns job result if successful (2 ms)
10:27:45
✓ returns job error if failed (2 ms)
10:27:45
✓ correctly returns job status for concurrent writes (2 ms)
10:27:45
✓ refuses stale jobs (39 ms)
10:27:45
✓ enqueues jobs (31 ms)
10:27:45
✓ ignores duplicate jobs (27 ms)
10:27:45
✓ reports correct status when enqueuing repeat jobs (32 ms)
10:27:45
✓ throws an error in case of duplicate job IDs (23 ms)
10:27:45
✓ returns not-found status for non-existing jobs (3 ms)
10:27:45
✓ cancels jobs in queue (28 ms)
10:27:45
✓ cancels jobs in-progress (28 ms)
10:27:45
✓ returns job result if successful (32 ms)
10:27:45
✓ returns job error if failed (41 ms)
10:27:45
✓ correctly returns job status for concurrent writes (44 ms)
10:27:45 Consumer API
10:27:45
✓ returns undefined if no jobs are available (7 ms)
10:27:45
✓ returns jobs in priority order (1 ms)
10:27:45
✓ returns undefined if no jobs are available for the given allowList (1 ms)
10:27:45
✓ returns a job if it is in the allowList (2 ms)
10:27:45
✓ returns the most important job if it is in the allowList (2 ms)
10:27:45
✓ returns any job if filter is empty (2 ms)
10:27:45
✓ returns a new job when reporting job success (2 ms)
10:27:45
✓ returns a new job when reporting permanent error (2 ms)
10:27:45
✓ returns a new job when reporting retry-able error (2 ms)
10:27:45
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
10:27:45
✓ returns a new job if job is already in progress elsewhere (1057 ms)
10:27:45
✓ avoids sending the same job to a new agent after a restart (1005 ms)
10:27:45
✓ avoids sending a completed job to a new agent after a restart (1007 ms)
10:27:45
✓ tracks job result if in progress (3 ms)
10:27:45
✓ tracks job result even if job is in queue (3 ms)
10:27:45
✓ ignores reported job error if unknown job (2 ms)
10:27:45
✓ ignores job result if unknown job (1 ms)
10:27:45
✓ returns undefined if no jobs are available (2 ms)
10:27:45
✓ returns jobs in priority order (53 ms)
10:27:45
✓ returns undefined if no jobs are available for the given allowList (22 ms)
10:27:45
✓ returns a job if it is in the allowList (73 ms)
10:27:45
✓ returns the most important job if it is in the allowList (66 ms)
10:27:45
✓ returns any job if filter is empty (72 ms)
10:27:45
✓ returns a new job when reporting job success (38 ms)
10:27:45
✓ returns a new job when reporting permanent error (69 ms)
10:27:45
✓ returns a new job when reporting retry-able error (46 ms)
10:27:45
✓ returns a new job when reporting progress if current one is cancelled (48 ms)
10:27:45
✓ returns a new job if job is already in progress elsewhere (1119 ms)
10:27:45
✓ avoids sending the same job to a new agent after a restart (1059 ms)
10:27:45
✓ avoids sending a completed job to a new agent after a restart (1081 ms)
10:27:45
✓ tracks job result if in progress (57 ms)
10:27:45
✓ tracks job result even if job is in queue (61 ms)
10:27:45
✓ ignores reported job error if unknown job (4 ms)
10:27:45
✓ ignores job result if unknown job (4 ms)
10:27:45 Timeouts
10:27:45
✓ tracks in progress jobs (1 ms)
10:27:45
✓ re-enqueues jobs that time out (108 ms)
10:27:45
✓ cancel stale jobs that time out (236 ms)
10:27:45
✓ keeps the jobs in progress while it is alive (207 ms)
10:27:45
✓ tracks in progress jobs (24 ms)
10:27:45
✓ re-enqueues jobs that time out (136 ms)
10:27:45
✓ cancel stale jobs that time out (288 ms)
10:27:45
✓ keeps the jobs in progress while it is alive (227 ms)
10:27:45 Retries
10:27:45
✓ retries jobs (4 ms)
10:27:45
✓ retries up to a maximum number of times (4 ms)
10:27:45
✓ passing retry=false does not retry (2 ms)
10:27:45
✓ does not retry if job is stale (29 ms)
10:27:45
✓ retries jobs (24 ms)
10:27:45
✓ retries up to a maximum number of times (31 ms)
10:27:45
✓ passing retry=false does not retry (38 ms)
10:27:45
✓ does not retry if job is stale (67 ms)
10:27:45 Database management
10:27:45
✓ re-enqueues proof requests on start (2 ms)
10:27:45
✓ restores proof results on start (2 ms)
10:27:45
✓ only re-enqueues unfinished jobs (1 ms)
10:27:45
✓ saves job when enqueued (2 ms)
10:27:45
✓ does not retain job if database fails to save (2 ms)
10:27:45
✓ saves job result (2 ms)
10:27:45
✓ saves result even if database fails to save (2 ms)
10:27:45
✓ saves job error (1 ms)
10:27:45
✓ saves job error even if database fails to save (2 ms)
10:27:45
✓ does not save job result if job is unknown (1 ms)
10:27:45
✓ does not save job error if job is unknown (1 ms)
10:27:45
✓ cleans up old jobs periodically (261 ms)
10:27:45
✓ re-enqueues proof requests on start (65 ms)
10:27:45
✓ restores proof results on start (68 ms)
10:27:45
✓ only re-enqueues unfinished jobs (53 ms)
10:27:45
✓ saves job when enqueued (30 ms)
10:27:45
✓ does not retain job if database fails to save (6 ms)
10:27:45
✓ saves job result (35 ms)
10:27:45
✓ saves result even if database fails to save (31 ms)
10:27:45
✓ saves job error (46 ms)
10:27:45
✓ saves job error even if database fails to save (28 ms)
10:27:45
✓ does not save job result if job is unknown (8 ms)
10:27:45
✓ does not save job error if job is unknown (6 ms)
10:27:45
✓ cleans up old jobs periodically (428 ms)
10:27:45
10:27:45
Test Suites: 1 passed, 1 total
10:27:45
Tests: 96 passed, 96 total
10:27:45
Snapshots: 0 total
10:27:45
Time: 13.351 s
10:27:45
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
10:27:45
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?