Parent Log:
http://ci.aztec-labs.com/806eea33511d2bcb
Command: dd02bf67482af96b yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/629a86eb64e58e40099493dcb61b8ddb08319fc5
Env: REF_NAME=gh-readonly-queue/next/pr-15327-dbfd8421af552f5a3c09615204514bbfb98a8a06 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Thu Jun 26 22:16:01 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15327_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
22:16:05 [22:16:05.633]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.639]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93aa26f333c17039 epochNumber=42
{"provingJobId":"1:BASE_PARITY:93aa26f333c17039"}
22:16:05 [22:16:05.642]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:c7bcfb74dd4c75e4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c7bcfb74dd4c75e4"}
22:16:05 [22:16:05.676]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.677]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d061b8ea6f95b39c epochNumber=1
{"provingJobId":"1:BASE_PARITY:d061b8ea6f95b39c"}
22:16:05 [22:16:05.678]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7e6b801148db605 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7e6b801148db605"}
22:16:05 [22:16:05.680]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.681]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4307450daa60e0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4307450daa60e0f"}
22:16:05 [22:16:05.682]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c4307450daa60e0f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c4307450daa60e0f"}
22:16:05 [22:16:05.684]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.685]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f6bd794a427c696 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f6bd794a427c696"}
22:16:05 [22:16:05.686]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9f6bd794a427c696 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9f6bd794a427c696"}
22:16:05 [22:16:05.686]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9f6bd794a427c696 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9f6bd794a427c696"}
22:16:05 [22:16:05.687]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9f6bd794a427c696 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9f6bd794a427c696"}
22:16:05 [22:16:05.688]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.689]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d287cfa679cdfaa epochNumber=1
{"provingJobId":"1:BASE_PARITY:2d287cfa679cdfaa"}
22:16:05 [22:16:05.713]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.714]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.715]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5332ee3e252fe0dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:5332ee3e252fe0dc"}
22:16:05 [22:16:05.715]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5332ee3e252fe0dc
{"provingJobId":"1:BASE_PARITY:5332ee3e252fe0dc"}
22:16:05 [22:16:05.716]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5332ee3e252fe0dc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5332ee3e252fe0dc"}
22:16:05 [22:16:05.716]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5332ee3e252fe0dc type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5332ee3e252fe0dc"}
22:16:05 [22:16:05.717]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.718]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52c85ac98ebf573d epochNumber=1
{"provingJobId":"1:BASE_PARITY:52c85ac98ebf573d"}
22:16:05 [22:16:05.718]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:52c85ac98ebf573d
{"provingJobId":"1:BASE_PARITY:52c85ac98ebf573d"}
22:16:05 [22:16:05.718]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:52c85ac98ebf573d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:52c85ac98ebf573d"}
22:16:05 [22:16:05.719]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.720]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cfb1107859c3bd69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cfb1107859c3bd69"}
22:16:05 [22:16:05.720]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cfb1107859c3bd69 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:cfb1107859c3bd69"}
22:16:05 [22:16:05.721]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cfb1107859c3bd69 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cfb1107859c3bd69"}
22:16:05 [22:16:05.722]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.722]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd7de4c423e7c7d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd7de4c423e7c7d0"}
22:16:05 [22:16:05.723]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd7de4c423e7c7d0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fd7de4c423e7c7d0"}
22:16:05 [22:16:05.723]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fd7de4c423e7c7d0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fd7de4c423e7c7d0"}
22:16:05 [22:16:05.724]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.724]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:988760e5d3faeb8d epochNumber=0
{"provingJobId":"1:BASE_PARITY:988760e5d3faeb8d"}
22:16:05 [22:16:05.725]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:988760e5d3faeb8d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:988760e5d3faeb8d"}
22:16:05 [22:16:05.725]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1eef4d5693be6e42 epochNumber=0
{"provingJobId":"1:BASE_PARITY:1eef4d5693be6e42"}
22:16:05 [22:16:05.725]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:988760e5d3faeb8d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:988760e5d3faeb8d"}
22:16:05 [22:16:05.726]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4aad0898c3a37af8 epochNumber=0
{"provingJobId":"1:BASE_PARITY:4aad0898c3a37af8"}
22:16:05 [22:16:05.735]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.736]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.737]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b319edfc0782a123 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b319edfc0782a123"}
22:16:05 [22:16:05.737]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fccda76a4f8a2933 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fccda76a4f8a2933"}
22:16:05 [22:16:05.738]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.739]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77ea13832ea173a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77ea13832ea173a7"}
22:16:05 [22:16:05.740]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ef16b5785c1376e epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ef16b5785c1376e"}
22:16:05 [22:16:05.741]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15ad377972fdb20d epochNumber=1
{"provingJobId":"1:BASE_PARITY:15ad377972fdb20d"}
22:16:05 [22:16:05.741]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f0faaeed9e2d951 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8f0faaeed9e2d951"}
22:16:05 [22:16:05.741]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4ff41c5cf730d61 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4ff41c5cf730d61"}
22:16:05 [22:16:05.742]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.743]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9c69214d06e5f12 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9c69214d06e5f12"}
22:16:05 [22:16:05.743]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54a1eaf0789ef109 epochNumber=1
{"provingJobId":"1:BASE_PARITY:54a1eaf0789ef109"}
22:16:05 [22:16:05.743]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf4d9d5e506f3bec epochNumber=2
{"provingJobId":"1:BASE_PARITY:cf4d9d5e506f3bec"}
22:16:05 [22:16:05.744]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a4f27803af7ebce epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a4f27803af7ebce"}
22:16:05 [22:16:05.745]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.745]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe6a8ba4957fd190 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe6a8ba4957fd190"}
22:16:05 [22:16:05.746]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68c9a910210dacd2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:68c9a910210dacd2"}
22:16:05 [22:16:05.746]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:204e715f40b11d28 epochNumber=2
{"provingJobId":"1:BASE_PARITY:204e715f40b11d28"}
22:16:05 [22:16:05.746]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85af8125c8ce97b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:85af8125c8ce97b3"}
22:16:05 [22:16:05.747]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.748]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c0f30d1c67d887e epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c0f30d1c67d887e"}
22:16:05 [22:16:05.748]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4086127a1545a5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4086127a1545a5d"}
22:16:05 [22:16:05.749]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3c0f30d1c67d887e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3c0f30d1c67d887e"}
22:16:05 [22:16:05.750]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.751]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12c765ed1e8aec56 epochNumber=1
{"provingJobId":"1:BASE_PARITY:12c765ed1e8aec56"}
22:16:05 [22:16:05.751]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:670aacbae5ad381f epochNumber=1
{"provingJobId":"1:BASE_PARITY:670aacbae5ad381f"}
22:16:05 [22:16:05.751]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:12c765ed1e8aec56 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:12c765ed1e8aec56"}
22:16:05 [22:16:05.752]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.753]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a92b01dda79d25ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:a92b01dda79d25ed"}
22:16:05 [22:16:05.753]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2fe17f63e84ae5a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2fe17f63e84ae5a4"}
22:16:05 [22:16:05.754]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a92b01dda79d25ed type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:a92b01dda79d25ed"}
22:16:05 [22:16:05.755]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.755]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93d35944bbc46ab5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:93d35944bbc46ab5"}
22:16:05 [22:16:05.756]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:93d35944bbc46ab5
{"provingJobId":"1:BASE_PARITY:93d35944bbc46ab5"}
22:16:05 [22:16:05.756]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:93d35944bbc46ab5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:93d35944bbc46ab5"}
22:16:05 [22:16:05.756]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c22199082fd2cdb8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c22199082fd2cdb8"}
22:16:05 [22:16:05.757]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:93d35944bbc46ab5 has already been completed
{"provingJobId":"1:BASE_PARITY:93d35944bbc46ab5"}
22:16:05 [22:16:05.758]
INFO:
prover-client:proving-broker Proving Broker started
22:16:05 [22:16:05.759]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e2e380d4074c82b epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e2e380d4074c82b"}
22:16:05 [22:16:05.759]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:449d93018c242f24 epochNumber=2
{"provingJobId":"1:BASE_PARITY:449d93018c242f24"}
22:16:06 [22:16:06.812]
INFO:
prover-client:proving-broker Proving Broker started
22:16:06 [22:16:06.813]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6e2e380d4074c82b settled=false
{"provingJobId":"1:BASE_PARITY:6e2e380d4074c82b","status":"pending"}
22:16:06 [22:16:06.813]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:449d93018c242f24 settled=false
{"provingJobId":"1:BASE_PARITY:449d93018c242f24","status":"pending"}
22:16:06 [22:16:06.814]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6e2e380d4074c82b type=BASE_PARITY startedAt=1750976165759 older agent has taken job
{"provingJobId":"1:BASE_PARITY:6e2e380d4074c82b"}
22:16:06 [22:16:06.815]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6e2e380d4074c82b type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:6e2e380d4074c82b"}
22:16:06 [22:16:06.816]
INFO:
prover-client:proving-broker Proving Broker started
22:16:06 [22:16:06.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99c5cd76efa89eb7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:99c5cd76efa89eb7"}
22:16:06 [22:16:06.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d30abd401ca9d8f epochNumber=2
{"provingJobId":"1:BASE_PARITY:4d30abd401ca9d8f"}
22:16:07 [22:16:07.817]
INFO:
prover-client:proving-broker Proving Broker started
22:16:07 [22:16:07.818]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:99c5cd76efa89eb7 settled=false
{"provingJobId":"1:BASE_PARITY:99c5cd76efa89eb7","status":"pending"}
22:16:07 [22:16:07.818]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4d30abd401ca9d8f settled=false
{"provingJobId":"1:BASE_PARITY:4d30abd401ca9d8f","status":"pending"}
22:16:07 [22:16:07.819]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:99c5cd76efa89eb7 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:99c5cd76efa89eb7"}
22:16:07 [22:16:07.820]
INFO:
prover-client:proving-broker Proving Broker started
22:16:07 [22:16:07.821]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60e2829a7ece562e epochNumber=1
{"provingJobId":"1:BASE_PARITY:60e2829a7ece562e"}
22:16:07 [22:16:07.821]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:27c0b18d93bbaaaa epochNumber=2
{"provingJobId":"1:BASE_PARITY:27c0b18d93bbaaaa"}
22:16:08 [22:16:08.822]
INFO:
prover-client:proving-broker Proving Broker started
22:16:08 [22:16:08.822]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:60e2829a7ece562e settled=false
{"provingJobId":"1:BASE_PARITY:60e2829a7ece562e","status":"pending"}
22:16:08 [22:16:08.823]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:27c0b18d93bbaaaa settled=false
{"provingJobId":"1:BASE_PARITY:27c0b18d93bbaaaa","status":"pending"}
22:16:08 [22:16:08.823]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:60e2829a7ece562e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:60e2829a7ece562e"}
22:16:08 [22:16:08.824]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:60e2829a7ece562e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:60e2829a7ece562e"}
22:16:08 [22:16:08.829]
INFO:
prover-client:proving-broker Proving Broker started
22:16:08 [22:16:08.830]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:901cccff31b030d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:901cccff31b030d4"}
22:16:08 [22:16:08.830]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f5a50121fceeb84 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1f5a50121fceeb84"}
22:16:08 [22:16:08.831]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:901cccff31b030d4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:901cccff31b030d4"}
22:16:08 [22:16:08.831]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1f5a50121fceeb84 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1f5a50121fceeb84"}
22:16:08 [22:16:08.833]
INFO:
prover-client:proving-broker Proving Broker started
22:16:08 [22:16:08.833]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24eb26895c6759ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:24eb26895c6759ac"}
22:16:08 [22:16:08.834]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc9b5cf21027efd7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cc9b5cf21027efd7"}
22:16:08 [22:16:08.834]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:24eb26895c6759ac type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:24eb26895c6759ac"}
22:16:08 [22:16:08.834]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:24eb26895c6759ac type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:24eb26895c6759ac"}
22:16:08 [22:16:08.835]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cc9b5cf21027efd7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:cc9b5cf21027efd7"}
22:16:08 [22:16:08.836]
INFO:
prover-client:proving-broker Proving Broker started
22:16:08 [22:16:08.836]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:248134772056a6fc err=test error
{"provingJoId":"1:BASE_PARITY:248134772056a6fc"}
22:16:08 [22:16:08.837]
INFO:
prover-client:proving-broker Proving Broker started
22:16:08 [22:16:08.838]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:66494e7824dcfb43 not found
{"provingJobId":"1:BASE_PARITY:66494e7824dcfb43"}
22:16:08 [22:16:08.839]
INFO:
prover-client:proving-broker Proving Broker started
22:16:08 [22:16:08.840]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:353a90245d1c3496 epochNumber=1
{"provingJobId":"1:BASE_PARITY:353a90245d1c3496"}
22:16:08 [22:16:08.841]
INFO:
prover-client:proving-broker Proving Broker started
22:16:08 [22:16:08.841]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7be23a506f4e5c90 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7be23a506f4e5c90"}
22:16:08 [22:16:08.942]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7be23a506f4e5c90 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:7be23a506f4e5c90"}
22:16:08 [22:16:08.945]
INFO:
prover-client:proving-broker Proving Broker started
22:16:08 [22:16:08.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e79f985f5192e05 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e79f985f5192e05"}
22:16:09 [22:16:09.071]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4e79f985f5192e05 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4e79f985f5192e05"}
22:16:09 [22:16:09.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f75ea768e66bc7ec epochNumber=10
{"provingJobId":"1:BASE_PARITY:f75ea768e66bc7ec"}
22:16:09 [22:16:09.097]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:09 [22:16:09.201]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.202]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ac8279a643ccf30 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1ac8279a643ccf30"}
22:16:09 [22:16:09.357]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1ac8279a643ccf30 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1ac8279a643ccf30"}
22:16:09 [22:16:09.407]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.408]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc43ec560c3a1a73 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc43ec560c3a1a73"}
22:16:09 [22:16:09.409]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cc43ec560c3a1a73 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:cc43ec560c3a1a73"}
22:16:09 [22:16:09.410]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.411]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8eba6ac7353ec423 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8eba6ac7353ec423"}
22:16:09 [22:16:09.411]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8eba6ac7353ec423 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:8eba6ac7353ec423"}
22:16:09 [22:16:09.412]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8eba6ac7353ec423 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:8eba6ac7353ec423"}
22:16:09 [22:16:09.413]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.414]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:800da961a6223006 epochNumber=1
{"provingJobId":"1:BASE_PARITY:800da961a6223006"}
22:16:09 [22:16:09.415]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:800da961a6223006 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:800da961a6223006"}
22:16:09 [22:16:09.416]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.416]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e45403931d5fadd2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e45403931d5fadd2"}
22:16:09 [22:16:09.417]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e45403931d5fadd2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:e45403931d5fadd2"}
22:16:09 [22:16:09.418]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22db4fb420b7cc39 epochNumber=3
{"provingJobId":"1:BASE_PARITY:22db4fb420b7cc39"}
22:16:09 [22:16:09.441]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:09 [22:16:09.442]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e45403931d5fadd2 err=test error
{"provingJoId":"1:BASE_PARITY:e45403931d5fadd2"}
22:16:09 [22:16:09.444]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.445]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:190e083e6b706204 settled=false
{"provingJobId":"1:BASE_PARITY:190e083e6b706204","status":"pending"}
22:16:09 [22:16:09.445]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cd1fa990b19d95cb settled=false
{"provingJobId":"1:BASE_PARITY:cd1fa990b19d95cb","status":"pending"}
22:16:09 [22:16:09.448]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.448]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:171dd4dc0d538045 settled=true
{"provingJobId":"1:BASE_PARITY:171dd4dc0d538045","status":"fulfilled"}
22:16:09 [22:16:09.448]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:6ae8342d5c4f885a settled=true
{"provingJobId":"2:BASE_PARITY:6ae8342d5c4f885a","status":"fulfilled"}
22:16:09 [22:16:09.450]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.450]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cd12be91034b860a settled=true
{"provingJobId":"1:BASE_PARITY:cd12be91034b860a","status":"fulfilled"}
22:16:09 [22:16:09.450]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4df3761313f8fd40 settled=false
{"provingJobId":"1:BASE_PARITY:4df3761313f8fd40","status":"pending"}
22:16:09 [22:16:09.451]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97fe14e74970ed0a epochNumber=1
{"provingJobId":"1:BASE_PARITY:97fe14e74970ed0a"}
22:16:09 [22:16:09.454]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.454]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2595efc4af2d96b epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2595efc4af2d96b"}
22:16:09 [22:16:09.455]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:b2595efc4af2d96b: Error: db error: Error: db error
22:16:09 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:b2595efc4af2d96b"}
22:16:09 [22:16:09.457]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.458]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e20fe5f0d4199601 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e20fe5f0d4199601"}
22:16:09 [22:16:09.458]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e20fe5f0d4199601 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e20fe5f0d4199601"}
22:16:09 [22:16:09.458]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e20fe5f0d4199601 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e20fe5f0d4199601"}
22:16:09 [22:16:09.459]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f764b02660fac83 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f764b02660fac83"}
22:16:09 [22:16:09.460]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5f764b02660fac83 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5f764b02660fac83"}
22:16:09 [22:16:09.460]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5f764b02660fac83 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5f764b02660fac83"}
22:16:09 [22:16:09.460]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:5f764b02660fac83: Error: db error
22:16:09 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:5f764b02660fac83"}
22:16:09 [22:16:09.461]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.462]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:329eee2170d5a961 epochNumber=1
{"provingJobId":"1:BASE_PARITY:329eee2170d5a961"}
22:16:09 [22:16:09.462]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:329eee2170d5a961 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:329eee2170d5a961"}
22:16:09 [22:16:09.462]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:329eee2170d5a961 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:329eee2170d5a961"}
22:16:09 [22:16:09.463]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.463]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:835d9d840800f130 epochNumber=1
{"provingJobId":"1:BASE_PARITY:835d9d840800f130"}
22:16:09 [22:16:09.463]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:835d9d840800f130 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:835d9d840800f130"}
22:16:09 [22:16:09.464]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:835d9d840800f130 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:835d9d840800f130"}
22:16:09 [22:16:09.464]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:835d9d840800f130 jobErr=test error: Error: db error
22:16:09 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:835d9d840800f130"}
22:16:09 [22:16:09.465]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.466]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dd771f491a0b1808 not found
{"provingJobId":"1:BASE_PARITY:dd771f491a0b1808"}
22:16:09 [22:16:09.467]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.467]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f8e84d9adcdf3b55 err=test error
{"provingJoId":"1:BASE_PARITY:f8e84d9adcdf3b55"}
22:16:09 [22:16:09.468]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.479]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8cb8055383f4a38b epochNumber=1
{"provingJobId":"1:BASE_PARITY:8cb8055383f4a38b"}
22:16:09 [22:16:09.480]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8cb8055383f4a38b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8cb8055383f4a38b"}
22:16:09 [22:16:09.480]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8cb8055383f4a38b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8cb8055383f4a38b"}
22:16:09 [22:16:09.490]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:e40e69d46909e991 epochNumber=2
{"provingJobId":"2:BASE_PARITY:e40e69d46909e991"}
22:16:09 [22:16:09.492]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:e40e69d46909e991 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:e40e69d46909e991"}
22:16:09 [22:16:09.494]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:e40e69d46909e991 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:e40e69d46909e991"}
22:16:09 [22:16:09.555]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:7b2380c97e9ba09e epochNumber=3
{"provingJobId":"3:BASE_PARITY:7b2380c97e9ba09e"}
22:16:09 [22:16:09.571]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:09 [22:16:09.620]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:fbd0bb82c8fbb7d4 epochNumber=4
{"provingJobId":"4:BASE_PARITY:fbd0bb82c8fbb7d4"}
22:16:09 [22:16:09.622]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:09 [22:16:09.685]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:3809028a5f6a6405 epochNumber=5
{"provingJobId":"5:BASE_PARITY:3809028a5f6a6405"}
22:16:09 [22:16:09.700]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:09 [22:16:09.742]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.743]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0582c11818bb67f2 epochNumber=42
{"provingJobId":"1:BASE_PARITY:0582c11818bb67f2"}
22:16:09 [22:16:09.745]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-test0XpOYP/42 with map size 1073741824
22:16:09 [22:16:09.753]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:09 [22:16:09.783]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:2334de15482cffd3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2334de15482cffd3"}
22:16:09 [22:16:09.791]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.792]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c31aaa0a6fe3e50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c31aaa0a6fe3e50"}
22:16:09 [22:16:09.793]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testy9tvCt/1 with map size 1073741824
22:16:09 [22:16:09.796]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:09 [22:16:09.817]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59ba70ab1d598faa epochNumber=1
{"provingJobId":"1:BASE_PARITY:59ba70ab1d598faa"}
22:16:09 [22:16:09.830]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.831]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77e7425f7edaf9b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77e7425f7edaf9b3"}
22:16:09 [22:16:09.832]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFzzVIR/1 with map size 1073741824
22:16:09 [22:16:09.833]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:09 [22:16:09.852]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:77e7425f7edaf9b3 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:77e7425f7edaf9b3"}
22:16:09 [22:16:09.858]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.860]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6d409862e895166 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6d409862e895166"}
22:16:09 [22:16:09.861]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3rs4Jt/1 with map size 1073741824
22:16:09 [22:16:09.862]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:09 [22:16:09.882]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c6d409862e895166 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c6d409862e895166"}
22:16:09 [22:16:09.884]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c6d409862e895166 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c6d409862e895166"}
22:16:09 [22:16:09.891]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c6d409862e895166 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c6d409862e895166"}
22:16:09 [22:16:09.899]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.900]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5060ab16c10439fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:5060ab16c10439fb"}
22:16:09 [22:16:09.901]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzOKQm7/1 with map size 1073741824
22:16:09 [22:16:09.904]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:09 [22:16:09.928]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.932]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.932]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:640071dd4bd151fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:640071dd4bd151fa"}
22:16:09 [22:16:09.934]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test488fJk/1 with map size 1073741824
22:16:09 [22:16:09.937]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:09 [22:16:09.955]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:640071dd4bd151fa
{"provingJobId":"1:BASE_PARITY:640071dd4bd151fa"}
22:16:09 [22:16:09.955]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:640071dd4bd151fa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:640071dd4bd151fa"}
22:16:09 [22:16:09.956]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:640071dd4bd151fa type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:640071dd4bd151fa"}
22:16:09 [22:16:09.966]
INFO:
prover-client:proving-broker Proving Broker started
22:16:09 [22:16:09.967]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8d06f4c8478ee16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8d06f4c8478ee16"}
22:16:09 [22:16:09.968]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiS6to0/1 with map size 1073741824
22:16:09 [22:16:09.974]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:09 [22:16:09.997]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b8d06f4c8478ee16
{"provingJobId":"1:BASE_PARITY:b8d06f4c8478ee16"}
22:16:09 [22:16:09.998]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b8d06f4c8478ee16 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b8d06f4c8478ee16"}
22:16:10 [22:16:10.006]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.007]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84261636ac6fa0cf epochNumber=1
{"provingJobId":"1:BASE_PARITY:84261636ac6fa0cf"}
22:16:10 [22:16:10.012]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYhBLx7/1 with map size 1073741824
22:16:10 [22:16:10.021]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.040]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:84261636ac6fa0cf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:84261636ac6fa0cf"}
22:16:10 [22:16:10.040]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:84261636ac6fa0cf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:84261636ac6fa0cf"}
22:16:10 [22:16:10.049]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.049]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f9f1e82976d9f2e epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f9f1e82976d9f2e"}
22:16:10 [22:16:10.050]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiNsRp6/1 with map size 1073741824
22:16:10 [22:16:10.052]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.070]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3f9f1e82976d9f2e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3f9f1e82976d9f2e"}
22:16:10 [22:16:10.071]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3f9f1e82976d9f2e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3f9f1e82976d9f2e"}
22:16:10 [22:16:10.080]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.081]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ce37d317e037925 epochNumber=0
{"provingJobId":"1:BASE_PARITY:9ce37d317e037925"}
22:16:10 [22:16:10.082]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test1v9Cik/0 with map size 1073741824
22:16:10 [22:16:10.083]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.102]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9ce37d317e037925 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9ce37d317e037925"}
22:16:10 [22:16:10.103]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca918fbbc23fe0f6 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ca918fbbc23fe0f6"}
22:16:10 [22:16:10.104]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9ce37d317e037925 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9ce37d317e037925"}
22:16:10 [22:16:10.104]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:849f7534326836fc epochNumber=0
{"provingJobId":"1:BASE_PARITY:849f7534326836fc"}
22:16:10 [22:16:10.122]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.126]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34d131aef17e91e2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:34d131aef17e91e2"}
22:16:10 [22:16:10.129]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJu2o66/2 with map size 1073741824
22:16:10 [22:16:10.132]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.156]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c149aee9246f7f36 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c149aee9246f7f36"}
22:16:10 [22:16:10.157]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJu2o66/1 with map size 1073741824
22:16:10 [22:16:10.160]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.181]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.182]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e2913582fda0950 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e2913582fda0950"}
22:16:10 [22:16:10.182]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxUtCln/1 with map size 1073741824
22:16:10 [22:16:10.184]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.207]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:abc2c43f50e80999 epochNumber=1
{"provingJobId":"1:BASE_PARITY:abc2c43f50e80999"}
22:16:10 [22:16:10.209]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtN0lwu/1 with map size 1073741824
22:16:10 [22:16:10.213]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:353e583e67acedfa epochNumber=1
{"provingJobId":"1:BASE_PARITY:353e583e67acedfa"}
22:16:10 [22:16:10.245]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9a6599894c9acc2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b9a6599894c9acc2"}
22:16:10 [22:16:10.247]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtN0lwu/2 with map size 1073741824
22:16:10 [22:16:10.250]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0628919bb39b54f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0628919bb39b54f9"}
22:16:10 [22:16:10.285]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.285]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d5540b91deb8b13 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2d5540b91deb8b13"}
22:16:10 [22:16:10.286]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testD6Fl9L/1 with map size 1073741824
22:16:10 [22:16:10.291]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.310]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc7c307977b5d507 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bc7c307977b5d507"}
22:16:10 [22:16:10.316]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b76e6beccedf260 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8b76e6beccedf260"}
22:16:10 [22:16:10.317]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testD6Fl9L/2 with map size 1073741824
22:16:10 [22:16:10.320]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.344]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:74fc10de1688c3d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:74fc10de1688c3d7"}
22:16:10 [22:16:10.354]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.355]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9f1099128f1ae89 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9f1099128f1ae89"}
22:16:10 [22:16:10.356]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUCXKXo/1 with map size 1073741824
22:16:10 [22:16:10.357]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.374]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40609fb435007541 epochNumber=1
{"provingJobId":"1:BASE_PARITY:40609fb435007541"}
22:16:10 [22:16:10.380]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:849aecca665976b2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:849aecca665976b2"}
22:16:10 [22:16:10.381]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUCXKXo/2 with map size 1073741824
22:16:10 [22:16:10.386]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.409]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca9906820d25bdbc epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca9906820d25bdbc"}
22:16:10 [22:16:10.423]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.424]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14b3454b60c5950a epochNumber=1
{"provingJobId":"1:BASE_PARITY:14b3454b60c5950a"}
22:16:10 [22:16:10.427]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvCx11i/1 with map size 1073741824
22:16:10 [22:16:10.429]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.455]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86e8fe32bb8a314e epochNumber=1
{"provingJobId":"1:BASE_PARITY:86e8fe32bb8a314e"}
22:16:10 [22:16:10.463]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:14b3454b60c5950a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:14b3454b60c5950a"}
22:16:10 [22:16:10.476]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.477]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6cb5698ba1dddf6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6cb5698ba1dddf6"}
22:16:10 [22:16:10.479]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvDAcnd/1 with map size 1073741824
22:16:10 [22:16:10.483]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.505]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4f55f993a752d01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4f55f993a752d01"}
22:16:10 [22:16:10.512]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a6cb5698ba1dddf6 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:a6cb5698ba1dddf6"}
22:16:10 [22:16:10.526]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ed9b3be6cef5cc5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ed9b3be6cef5cc5"}
22:16:10 [22:16:10.528]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testE2Rl5j/1 with map size 1073741824
22:16:10 [22:16:10.534]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.555]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f0b8967f624b650 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f0b8967f624b650"}
22:16:10 [22:16:10.562]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2ed9b3be6cef5cc5 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:2ed9b3be6cef5cc5"}
22:16:10 [22:16:10.565]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.566]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8dda5d126630394 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f8dda5d126630394"}
22:16:10 [22:16:10.567]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFzefCP/1 with map size 1073741824
22:16:10 [22:16:10.571]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.593]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f8dda5d126630394
{"provingJobId":"1:BASE_PARITY:f8dda5d126630394"}
22:16:10 [22:16:10.594]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f8dda5d126630394 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f8dda5d126630394"}
22:16:10 [22:16:10.600]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a2ea824ce04f9bc epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a2ea824ce04f9bc"}
22:16:10 [22:16:10.607]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f8dda5d126630394 has already been completed
{"provingJobId":"1:BASE_PARITY:f8dda5d126630394"}
22:16:10 [22:16:10.611]
INFO:
prover-client:proving-broker Proving Broker started
22:16:10 [22:16:10.612]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ae836dd0cc268c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ae836dd0cc268c3"}
22:16:10 [22:16:10.614]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiHuDlY/1 with map size 1073741824
22:16:10 [22:16:10.616]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:10 [22:16:10.640]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84afc2cf5477fa25 epochNumber=2
{"provingJobId":"1:BASE_PARITY:84afc2cf5477fa25"}
22:16:10 [22:16:10.644]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testiHuDlY/2 with map size 1073741824
22:16:10 [22:16:10.648]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:11 [22:16:11.722]
INFO:
prover-client:proving-broker Proving Broker started
22:16:11 [22:16:11.730]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6ae836dd0cc268c3 settled=false
{"provingJobId":"1:BASE_PARITY:6ae836dd0cc268c3","status":"pending"}
22:16:11 [22:16:11.733]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:84afc2cf5477fa25 settled=false
{"provingJobId":"1:BASE_PARITY:84afc2cf5477fa25","status":"pending"}
22:16:11 [22:16:11.735]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6ae836dd0cc268c3 type=BASE_PARITY startedAt=1750976170670 older agent has taken job
{"provingJobId":"1:BASE_PARITY:6ae836dd0cc268c3"}
22:16:11 [22:16:11.736]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6ae836dd0cc268c3 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:6ae836dd0cc268c3"}
22:16:11 [22:16:11.741]
INFO:
prover-client:proving-broker Proving Broker started
22:16:11 [22:16:11.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd0d9b4507afc767 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd0d9b4507afc767"}
22:16:11 [22:16:11.743]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testX6AYda/1 with map size 1073741824
22:16:11 [22:16:11.746]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:11 [22:16:11.764]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f04906e6ed0fc5b7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f04906e6ed0fc5b7"}
22:16:11 [22:16:11.765]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testX6AYda/2 with map size 1073741824
22:16:11 [22:16:11.768]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:12 [22:16:12.786]
INFO:
prover-client:proving-broker Proving Broker started
22:16:12 [22:16:12.788]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fd0d9b4507afc767 settled=false
{"provingJobId":"1:BASE_PARITY:fd0d9b4507afc767","status":"pending"}
22:16:12 [22:16:12.789]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f04906e6ed0fc5b7 settled=false
{"provingJobId":"1:BASE_PARITY:f04906e6ed0fc5b7","status":"pending"}
22:16:12 [22:16:12.790]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd0d9b4507afc767 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:fd0d9b4507afc767"}
22:16:12 [22:16:12.793]
INFO:
prover-client:proving-broker Proving Broker started
22:16:12 [22:16:12.794]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67be1699123dd66d epochNumber=1
{"provingJobId":"1:BASE_PARITY:67be1699123dd66d"}
22:16:12 [22:16:12.795]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYqvE43/1 with map size 1073741824
22:16:12 [22:16:12.796]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:12 [22:16:12.819]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d73d6d58eb886c9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3d73d6d58eb886c9"}
22:16:12 [22:16:12.820]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYqvE43/2 with map size 1073741824
22:16:12 [22:16:12.826]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:13 [22:16:13.851]
INFO:
prover-client:proving-broker Proving Broker started
22:16:13 [22:16:13.853]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:67be1699123dd66d settled=false
{"provingJobId":"1:BASE_PARITY:67be1699123dd66d","status":"pending"}
22:16:13 [22:16:13.855]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3d73d6d58eb886c9 settled=false
{"provingJobId":"1:BASE_PARITY:3d73d6d58eb886c9","status":"pending"}
22:16:13 [22:16:13.857]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:67be1699123dd66d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:67be1699123dd66d"}
22:16:13 [22:16:13.857]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:67be1699123dd66d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:67be1699123dd66d"}
22:16:13 [22:16:13.867]
INFO:
prover-client:proving-broker Proving Broker started
22:16:13 [22:16:13.868]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d086dbd6206f1135 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d086dbd6206f1135"}
22:16:13 [22:16:13.870]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpHuNms/1 with map size 1073741824
22:16:13 [22:16:13.877]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:13 [22:16:13.899]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b05be0fff955bc3d epochNumber=2
{"provingJobId":"1:BASE_PARITY:b05be0fff955bc3d"}
22:16:13 [22:16:13.900]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpHuNms/2 with map size 1073741824
22:16:13 [22:16:13.902]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:13 [22:16:13.918]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d086dbd6206f1135 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d086dbd6206f1135"}
22:16:13 [22:16:13.924]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b05be0fff955bc3d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b05be0fff955bc3d"}
22:16:13 [22:16:13.934]
INFO:
prover-client:proving-broker Proving Broker started
22:16:13 [22:16:13.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40e52a3c4792e81f epochNumber=1
{"provingJobId":"1:BASE_PARITY:40e52a3c4792e81f"}
22:16:13 [22:16:13.936]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTEM1Ti/1 with map size 1073741824
22:16:13 [22:16:13.940]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:13 [22:16:13.957]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:440ff561c500b61b epochNumber=2
{"provingJobId":"1:BASE_PARITY:440ff561c500b61b"}
22:16:13 [22:16:13.958]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTEM1Ti/2 with map size 1073741824
22:16:13 [22:16:13.960]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:13 [22:16:13.975]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:40e52a3c4792e81f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:40e52a3c4792e81f"}
22:16:13 [22:16:13.975]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:40e52a3c4792e81f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:40e52a3c4792e81f"}
22:16:13 [22:16:13.982]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:440ff561c500b61b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:440ff561c500b61b"}
22:16:13 [22:16:13.991]
INFO:
prover-client:proving-broker Proving Broker started
22:16:13 [22:16:13.992]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8a1ad89fa1114306 err=test error
{"provingJoId":"1:BASE_PARITY:8a1ad89fa1114306"}
22:16:13 [22:16:13.994]
INFO:
prover-client:proving-broker Proving Broker started
22:16:13 [22:16:13.995]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:29e62391dbfbde2f not found
{"provingJobId":"1:BASE_PARITY:29e62391dbfbde2f"}
22:16:13 [22:16:13.997]
INFO:
prover-client:proving-broker Proving Broker started
22:16:13 [22:16:13.998]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f8c8b77fa6428db epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f8c8b77fa6428db"}
22:16:13 [22:16:13.998]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testN6wodx/1 with map size 1073741824
22:16:13 [22:16:13.999]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.024]
INFO:
prover-client:proving-broker Proving Broker started
22:16:14 [22:16:14.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d9bb29fe4413597 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d9bb29fe4413597"}
22:16:14 [22:16:14.026]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxNIi3K/1 with map size 1073741824
22:16:14 [22:16:14.027]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.151]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8d9bb29fe4413597 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8d9bb29fe4413597"}
22:16:14 [22:16:14.159]
INFO:
prover-client:proving-broker Proving Broker started
22:16:14 [22:16:14.160]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47397e21afff6c78 epochNumber=1
{"provingJobId":"1:BASE_PARITY:47397e21afff6c78"}
22:16:14 [22:16:14.160]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYJMoQ0/1 with map size 1073741824
22:16:14 [22:16:14.162]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.285]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:47397e21afff6c78 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:47397e21afff6c78"}
22:16:14 [22:16:14.290]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c83160d457b563c epochNumber=10
{"provingJobId":"1:BASE_PARITY:1c83160d457b563c"}
22:16:14 [22:16:14.291]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testYJMoQ0/10 with map size 1073741824
22:16:14 [22:16:14.293]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.337]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:14 [22:16:14.338]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
22:16:14 [22:16:14.444]
INFO:
prover-client:proving-broker Proving Broker started
22:16:14 [22:16:14.445]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01b7ced41c88c098 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01b7ced41c88c098"}
22:16:14 [22:16:14.446]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJaMOEU/1 with map size 1073741824
22:16:14 [22:16:14.450]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.648]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:01b7ced41c88c098 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:01b7ced41c88c098"}
22:16:14 [22:16:14.682]
INFO:
prover-client:proving-broker Proving Broker started
22:16:14 [22:16:14.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5dc4ff1c7c2b947d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5dc4ff1c7c2b947d"}
22:16:14 [22:16:14.684]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXdqrGD/1 with map size 1073741824
22:16:14 [22:16:14.687]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.707]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5dc4ff1c7c2b947d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:5dc4ff1c7c2b947d"}
22:16:14 [22:16:14.713]
INFO:
prover-client:proving-broker Proving Broker started
22:16:14 [22:16:14.714]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:240ce90d7e78b444 epochNumber=1
{"provingJobId":"1:BASE_PARITY:240ce90d7e78b444"}
22:16:14 [22:16:14.715]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIRYLWj/1 with map size 1073741824
22:16:14 [22:16:14.718]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.739]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:240ce90d7e78b444 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:240ce90d7e78b444"}
22:16:14 [22:16:14.740]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:240ce90d7e78b444 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:240ce90d7e78b444"}
22:16:14 [22:16:14.751]
INFO:
prover-client:proving-broker Proving Broker started
22:16:14 [22:16:14.753]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b946e97c68d2bba1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b946e97c68d2bba1"}
22:16:14 [22:16:14.754]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNAktkT/1 with map size 1073741824
22:16:14 [22:16:14.759]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.784]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b946e97c68d2bba1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b946e97c68d2bba1"}
22:16:14 [22:16:14.793]
INFO:
prover-client:proving-broker Proving Broker started
22:16:14 [22:16:14.794]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1271074b6b89d664 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1271074b6b89d664"}
22:16:14 [22:16:14.794]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDOf0by/1 with map size 1073741824
22:16:14 [22:16:14.796]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.815]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1271074b6b89d664 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1271074b6b89d664"}
22:16:14 [22:16:14.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b84e6fb7cfdf1c9 epochNumber=3
{"provingJobId":"1:BASE_PARITY:6b84e6fb7cfdf1c9"}
22:16:14 [22:16:14.818]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testDOf0by/3 with map size 1073741824
22:16:14 [22:16:14.821]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.870]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:14 [22:16:14.871]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
22:16:14 [22:16:14.871]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1271074b6b89d664 err=test error
{"provingJoId":"1:BASE_PARITY:1271074b6b89d664"}
22:16:14 [22:16:14.878]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhsAk90/1 with map size 1073741824
22:16:14 [22:16:14.883]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.902]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testhsAk90/2 with map size 1073741824
22:16:14 [22:16:14.905]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.925]
INFO:
prover-client:proving-broker Proving Broker started
22:16:14 [22:16:14.927]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b5546773436a17a3 settled=false
{"provingJobId":"1:BASE_PARITY:b5546773436a17a3","status":"pending"}
22:16:14 [22:16:14.929]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5f7242464be6001e settled=false
{"provingJobId":"1:BASE_PARITY:5f7242464be6001e","status":"pending"}
22:16:14 [22:16:14.944]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjAwkAd/1 with map size 1073741824
22:16:14 [22:16:14.947]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.964]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjAwkAd/2 with map size 1073741824
22:16:14 [22:16:14.966]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:14 [22:16:14.993]
INFO:
prover-client:proving-broker Proving Broker started
22:16:14 [22:16:14.995]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0f62f88e11f41568 settled=true
{"provingJobId":"1:BASE_PARITY:0f62f88e11f41568","status":"fulfilled"}
22:16:14 [22:16:14.997]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:870f6e28c0ee9e5c settled=true
{"provingJobId":"2:BASE_PARITY:870f6e28c0ee9e5c","status":"fulfilled"}
22:16:15 [22:16:15.001]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnF92B0/1 with map size 1073741824
22:16:15 [22:16:15.003]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.036]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnF92B0/2 with map size 1073741824
22:16:15 [22:16:15.041]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.064]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.066]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7ddaa6bd4a75f0d9 settled=true
{"provingJobId":"1:BASE_PARITY:7ddaa6bd4a75f0d9","status":"fulfilled"}
22:16:15 [22:16:15.068]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c923a19391422b1e settled=false
{"provingJobId":"1:BASE_PARITY:c923a19391422b1e","status":"pending"}
22:16:15 [22:16:15.073]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09b8ae7c767aa386 epochNumber=1
{"provingJobId":"1:BASE_PARITY:09b8ae7c767aa386"}
22:16:15 [22:16:15.075]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxjcrWy/1 with map size 1073741824
22:16:15 [22:16:15.080]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.112]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.112]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fd0112aaf65c909 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fd0112aaf65c909"}
22:16:15 [22:16:15.113]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:4fd0112aaf65c909: Error: db error: Error: db error
22:16:15 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:4fd0112aaf65c909"}
22:16:15 [22:16:15.116]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.116]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:110b026cd987d07e epochNumber=1
{"provingJobId":"1:BASE_PARITY:110b026cd987d07e"}
22:16:15 [22:16:15.118]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAlSJAY/1 with map size 1073741824
22:16:15 [22:16:15.121]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.149]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:110b026cd987d07e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:110b026cd987d07e"}
22:16:15 [22:16:15.149]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:110b026cd987d07e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:110b026cd987d07e"}
22:16:15 [22:16:15.165]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.166]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:445ee224ff6dbdab epochNumber=1
{"provingJobId":"1:BASE_PARITY:445ee224ff6dbdab"}
22:16:15 [22:16:15.167]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrP6izD/1 with map size 1073741824
22:16:15 [22:16:15.173]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.199]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:445ee224ff6dbdab type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:445ee224ff6dbdab"}
22:16:15 [22:16:15.200]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:445ee224ff6dbdab type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:445ee224ff6dbdab"}
22:16:15 [22:16:15.202]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:445ee224ff6dbdab: Error: db error
22:16:15 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:445ee224ff6dbdab"}
22:16:15 [22:16:15.206]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.207]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:431da8cf2c0f7904 epochNumber=1
{"provingJobId":"1:BASE_PARITY:431da8cf2c0f7904"}
22:16:15 [22:16:15.209]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGyseV4/1 with map size 1073741824
22:16:15 [22:16:15.216]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.239]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:431da8cf2c0f7904 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:431da8cf2c0f7904"}
22:16:15 [22:16:15.240]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:431da8cf2c0f7904 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:431da8cf2c0f7904"}
22:16:15 [22:16:15.255]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d88b71607ca30afc epochNumber=1
{"provingJobId":"1:BASE_PARITY:d88b71607ca30afc"}
22:16:15 [22:16:15.257]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyWW9xZ/1 with map size 1073741824
22:16:15 [22:16:15.260]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.289]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d88b71607ca30afc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d88b71607ca30afc"}
22:16:15 [22:16:15.289]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d88b71607ca30afc type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d88b71607ca30afc"}
22:16:15 [22:16:15.290]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:d88b71607ca30afc jobErr=test error: Error: db error
22:16:15 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:d88b71607ca30afc"}
22:16:15 [22:16:15.295]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.296]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:21dee7f611e80d02 not found
{"provingJobId":"1:BASE_PARITY:21dee7f611e80d02"}
22:16:15 [22:16:15.304]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.305]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bb00e728473a02a1 err=test error
{"provingJoId":"1:BASE_PARITY:bb00e728473a02a1"}
22:16:15 [22:16:15.314]
INFO:
prover-client:proving-broker Proving Broker started
22:16:15 [22:16:15.329]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c260bd31c98eb88 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c260bd31c98eb88"}
22:16:15 [22:16:15.331]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4nApzR/1 with map size 1073741824
22:16:15 [22:16:15.335]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.357]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9c260bd31c98eb88 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9c260bd31c98eb88"}
22:16:15 [22:16:15.358]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9c260bd31c98eb88 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9c260bd31c98eb88"}
22:16:15 [22:16:15.376]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:3a275b333bbc1eee epochNumber=2
{"provingJobId":"2:BASE_PARITY:3a275b333bbc1eee"}
22:16:15 [22:16:15.378]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4nApzR/2 with map size 1073741824
22:16:15 [22:16:15.383]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.404]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:3a275b333bbc1eee type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:3a275b333bbc1eee"}
22:16:15 [22:16:15.405]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:3a275b333bbc1eee type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:3a275b333bbc1eee"}
22:16:15 [22:16:15.474]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:fdd82a11188e50db epochNumber=3
{"provingJobId":"3:BASE_PARITY:fdd82a11188e50db"}
22:16:15 [22:16:15.476]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test4nApzR/3 with map size 1073741824
22:16:15 [22:16:15.479]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.518]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:15 [22:16:15.518]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
22:16:15 [22:16:15.562]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:7e28b989f72255c4 epochNumber=4
{"provingJobId":"4:BASE_PARITY:7e28b989f72255c4"}
22:16:15 [22:16:15.564]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-test4nApzR/4 with map size 1073741824
22:16:15 [22:16:15.570]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.599]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:15 [22:16:15.599]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
22:16:15 [22:16:15.648]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:a3ebcd7f49e3aeef epochNumber=5
{"provingJobId":"5:BASE_PARITY:a3ebcd7f49e3aeef"}
22:16:15 [22:16:15.650]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-test4nApzR/5 with map size 1073741824
22:16:15 [22:16:15.653]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
22:16:15 [22:16:15.681]
INFO:
prover-client:proving-broker Cleaned up jobs=1
22:16:15 [22:16:15.681]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
22:16:15
PASS src/proving_broker/proving_broker.test.ts (
13.411 s)
22:16:15 ProvingBroker
22:16:15 Producer API
22:16:15
✓ refuses stale jobs (44 ms)
22:16:15
✓ enqueues jobs (4 ms)
22:16:15
✓ ignores duplicate jobs (4 ms)
22:16:15
✓ reports correct status when enqueuing repeat jobs (5 ms)
22:16:15
✓ throws an error in case of duplicate job IDs (24 ms)
22:16:15
✓ returns not-found status for non-existing jobs (2 ms)
22:16:15
✓ cancels jobs in queue (3 ms)
22:16:15
✓ cancels jobs in-progress (2 ms)
22:16:15
✓ returns job result if successful (2 ms)
22:16:15
✓ returns job error if failed (2 ms)
22:16:15
✓ correctly returns job status for concurrent writes (10 ms)
22:16:15
✓ refuses stale jobs (51 ms)
22:16:15
✓ enqueues jobs (37 ms)
22:16:15
✓ ignores duplicate jobs (29 ms)
22:16:15
✓ reports correct status when enqueuing repeat jobs (39 ms)
22:16:15
✓ throws an error in case of duplicate job IDs (31 ms)
22:16:15
✓ returns not-found status for non-existing jobs (3 ms)
22:16:15
✓ cancels jobs in queue (34 ms)
22:16:15
✓ cancels jobs in-progress (40 ms)
22:16:15
✓ returns job result if successful (41 ms)
22:16:15
✓ returns job error if failed (31 ms)
22:16:15
✓ correctly returns job status for concurrent writes (42 ms)
22:16:15 Consumer API
22:16:15
✓ returns undefined if no jobs are available (2 ms)
22:16:15
✓ returns jobs in priority order (2 ms)
22:16:15
✓ returns undefined if no jobs are available for the given allowList (1 ms)
22:16:15
✓ returns a job if it is in the allowList (3 ms)
22:16:15
✓ returns the most important job if it is in the allowList (2 ms)
22:16:15
✓ returns any job if filter is empty (3 ms)
22:16:15
✓ returns a new job when reporting job success (3 ms)
22:16:15
✓ returns a new job when reporting permanent error (2 ms)
22:16:15
✓ returns a new job when reporting retry-able error (2 ms)
22:16:15
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
22:16:15
✓ returns a new job if job is already in progress elsewhere (1057 ms)
22:16:15
✓ avoids sending the same job to a new agent after a restart (1005 ms)
22:16:15
✓ avoids sending a completed job to a new agent after a restart (1004 ms)
22:16:15
✓ tracks job result if in progress (7 ms)
22:16:15
✓ tracks job result even if job is in queue (3 ms)
22:16:15
✓ ignores reported job error if unknown job (2 ms)
22:16:15
✓ ignores job result if unknown job (1 ms)
22:16:15
✓ returns undefined if no jobs are available (3 ms)
22:16:15
✓ returns jobs in priority order (55 ms)
22:16:15
✓ returns undefined if no jobs are available for the given allowList (26 ms)
22:16:15
✓ returns a job if it is in the allowList (76 ms)
22:16:15
✓ returns the most important job if it is in the allowList (69 ms)
22:16:15
✓ returns any job if filter is empty (67 ms)
22:16:15
✓ returns a new job when reporting job success (53 ms)
22:16:15
✓ returns a new job when reporting permanent error (46 ms)
22:16:15
✓ returns a new job when reporting retry-able error (42 ms)
22:16:15
✓ returns a new job when reporting progress if current one is cancelled (45 ms)
22:16:15
✓ returns a new job if job is already in progress elsewhere (1129 ms)
22:16:15
✓ avoids sending the same job to a new agent after a restart (1052 ms)
22:16:15
✓ avoids sending a completed job to a new agent after a restart (1073 ms)
22:16:15
✓ tracks job result if in progress (67 ms)
22:16:15
✓ tracks job result even if job is in queue (56 ms)
22:16:15
✓ ignores reported job error if unknown job (3 ms)
22:16:15
✓ ignores job result if unknown job (2 ms)
22:16:15 Timeouts
22:16:15
✓ tracks in progress jobs (1 ms)
22:16:15
✓ re-enqueues jobs that time out (103 ms)
22:16:15
✓ cancel stale jobs that time out (255 ms)
22:16:15
✓ keeps the jobs in progress while it is alive (205 ms)
22:16:15
✓ tracks in progress jobs (26 ms)
22:16:15
✓ re-enqueues jobs that time out (133 ms)
22:16:15
✓ cancel stale jobs that time out (287 ms)
22:16:15
✓ keeps the jobs in progress while it is alive (236 ms)
22:16:15 Retries
22:16:15
✓ retries jobs (3 ms)
22:16:15
✓ retries up to a maximum number of times (3 ms)
22:16:15
✓ passing retry=false does not retry (2 ms)
22:16:15
✓ does not retry if job is stale (28 ms)
22:16:15
✓ retries jobs (29 ms)
22:16:15
✓ retries up to a maximum number of times (39 ms)
22:16:15
✓ passing retry=false does not retry (42 ms)
22:16:15
✓ does not retry if job is stale (82 ms)
22:16:15 Database management
22:16:15
✓ re-enqueues proof requests on start (4 ms)
22:16:15
✓ restores proof results on start (2 ms)
22:16:15
✓ only re-enqueues unfinished jobs (2 ms)
22:16:15
✓ saves job when enqueued (2 ms)
22:16:15
✓ does not retain job if database fails to save (2 ms)
22:16:15
✓ saves job result (2 ms)
22:16:15
✓ saves result even if database fails to save (2 ms)
22:16:15
✓ saves job error (1 ms)
22:16:15
✓ saves job error even if database fails to save (2 ms)
22:16:15
✓ does not save job result if job is unknown (1 ms)
22:16:15
✓ does not save job error if job is unknown (1 ms)
22:16:15
✓ cleans up old jobs periodically (269 ms)
22:16:15
✓ re-enqueues proof requests on start (60 ms)
22:16:15
✓ restores proof results on start (64 ms)
22:16:15
✓ only re-enqueues unfinished jobs (72 ms)
22:16:15
✓ saves job when enqueued (37 ms)
22:16:15
✓ does not retain job if database fails to save (5 ms)
22:16:15
✓ saves job result (46 ms)
22:16:15
✓ saves result even if database fails to save (44 ms)
22:16:15
✓ saves job error (47 ms)
22:16:15
✓ saves job error even if database fails to save (40 ms)
22:16:15
✓ does not save job result if job is unknown (7 ms)
22:16:15
✓ does not save job error if job is unknown (8 ms)
22:16:15
✓ cleans up old jobs periodically (422 ms)
22:16:15
22:16:15
Test Suites: 1 passed, 1 total
22:16:15
Tests: 96 passed, 96 total
22:16:15
Snapshots: 0 total
22:16:15
Time: 13.497 s
22:16:15
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
22:16:15
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?