Parent Log:
http://ci.aztec-labs.com/af8b5f0ffc64dcdd
Command: 9b90ba1cd25aa9c9 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/4ae29efe4b68ec58d68cf3708b9a36b94dad8b2b
Env: REF_NAME=gh-readonly-queue/next/pr-15129-5c77169d7f01a9e15bc19b05fa231fd96111d7dd CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Wed Jun 18 15:19:20 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15129_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
15:19:24 [15:19:24.692]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.699]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d90b40f5cddadc5a epochNumber=42
{"provingJobId":"1:BASE_PARITY:d90b40f5cddadc5a"}
15:19:24 [15:19:24.702]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:52a6ff56e0ffefad epochNumber=1
{"provingJobId":"1:BASE_PARITY:52a6ff56e0ffefad"}
15:19:24 [15:19:24.739]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d17a385bae65bc11 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d17a385bae65bc11"}
15:19:24 [15:19:24.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d21aceceb49ace89 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d21aceceb49ace89"}
15:19:24 [15:19:24.743]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.744]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cfde160baa4bd916 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cfde160baa4bd916"}
15:19:24 [15:19:24.745]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cfde160baa4bd916 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:cfde160baa4bd916"}
15:19:24 [15:19:24.747]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.747]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f378471ad6d27772 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f378471ad6d27772"}
15:19:24 [15:19:24.749]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f378471ad6d27772 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f378471ad6d27772"}
15:19:24 [15:19:24.749]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f378471ad6d27772 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f378471ad6d27772"}
15:19:24 [15:19:24.750]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f378471ad6d27772 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f378471ad6d27772"}
15:19:24 [15:19:24.751]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.751]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ad038d2fde93cdb4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ad038d2fde93cdb4"}
15:19:24 [15:19:24.777]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.779]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.779]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10d540f4e7f92a6b epochNumber=1
{"provingJobId":"1:BASE_PARITY:10d540f4e7f92a6b"}
15:19:24 [15:19:24.780]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:10d540f4e7f92a6b
{"provingJobId":"1:BASE_PARITY:10d540f4e7f92a6b"}
15:19:24 [15:19:24.780]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:10d540f4e7f92a6b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:10d540f4e7f92a6b"}
15:19:24 [15:19:24.781]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:10d540f4e7f92a6b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:10d540f4e7f92a6b"}
15:19:24 [15:19:24.782]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20e6f3c88470a8e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:20e6f3c88470a8e2"}
15:19:24 [15:19:24.783]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:20e6f3c88470a8e2
{"provingJobId":"1:BASE_PARITY:20e6f3c88470a8e2"}
15:19:24 [15:19:24.783]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:20e6f3c88470a8e2 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:20e6f3c88470a8e2"}
15:19:24 [15:19:24.784]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec81f3e58149de9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec81f3e58149de9f"}
15:19:24 [15:19:24.785]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ec81f3e58149de9f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ec81f3e58149de9f"}
15:19:24 [15:19:24.786]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ec81f3e58149de9f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ec81f3e58149de9f"}
15:19:24 [15:19:24.787]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.787]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20ebb88059886b03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:20ebb88059886b03"}
15:19:24 [15:19:24.788]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:20ebb88059886b03 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:20ebb88059886b03"}
15:19:24 [15:19:24.788]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:20ebb88059886b03 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:20ebb88059886b03"}
15:19:24 [15:19:24.789]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.789]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:913285f8b03da9b4 epochNumber=0
{"provingJobId":"1:BASE_PARITY:913285f8b03da9b4"}
15:19:24 [15:19:24.789]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:913285f8b03da9b4 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:913285f8b03da9b4"}
15:19:24 [15:19:24.790]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81657f989a844337 epochNumber=0
{"provingJobId":"1:BASE_PARITY:81657f989a844337"}
15:19:24 [15:19:24.790]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:913285f8b03da9b4 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:913285f8b03da9b4"}
15:19:24 [15:19:24.790]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17c79cabedc7684f epochNumber=0
{"provingJobId":"1:BASE_PARITY:17c79cabedc7684f"}
15:19:24 [15:19:24.791]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.800]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.801]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e58ee76e904c0a48 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e58ee76e904c0a48"}
15:19:24 [15:19:24.801]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ac15b084d33a12d epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ac15b084d33a12d"}
15:19:24 [15:19:24.802]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.803]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d78ffb27957d22c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d78ffb27957d22c0"}
15:19:24 [15:19:24.804]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eade0da3138a0baf epochNumber=1
{"provingJobId":"1:BASE_PARITY:eade0da3138a0baf"}
15:19:24 [15:19:24.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92bebd342d34fc2c epochNumber=1
{"provingJobId":"1:BASE_PARITY:92bebd342d34fc2c"}
15:19:24 [15:19:24.805]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ba58f463fea206c epochNumber=2
{"provingJobId":"1:BASE_PARITY:5ba58f463fea206c"}
15:19:24 [15:19:24.805]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9d6551378b66aa6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a9d6551378b66aa6"}
15:19:24 [15:19:24.806]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86fabcac4a91c6b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:86fabcac4a91c6b8"}
15:19:24 [15:19:24.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2732dc1c3a41fb7b epochNumber=1
{"provingJobId":"1:BASE_PARITY:2732dc1c3a41fb7b"}
15:19:24 [15:19:24.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf61d8102489b5b7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cf61d8102489b5b7"}
15:19:24 [15:19:24.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a9e3276dba01abe epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a9e3276dba01abe"}
15:19:24 [15:19:24.808]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:504e3a144293f7f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:504e3a144293f7f3"}
15:19:24 [15:19:24.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a32f2fe00a16c66c epochNumber=1
{"provingJobId":"1:BASE_PARITY:a32f2fe00a16c66c"}
15:19:24 [15:19:24.809]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45b0940d9f7362a7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:45b0940d9f7362a7"}
15:19:24 [15:19:24.809]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02fd51a11c534563 epochNumber=1
{"provingJobId":"1:BASE_PARITY:02fd51a11c534563"}
15:19:24 [15:19:24.811]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.814]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bbad215bb32271d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bbad215bb32271d1"}
15:19:24 [15:19:24.814]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:199ee27e32d275c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:199ee27e32d275c7"}
15:19:24 [15:19:24.815]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bbad215bb32271d1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bbad215bb32271d1"}
15:19:24 [15:19:24.816]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.817]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91b6b4bd9a001b4e epochNumber=1
{"provingJobId":"1:BASE_PARITY:91b6b4bd9a001b4e"}
15:19:24 [15:19:24.817]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:18c5fa121274369d epochNumber=1
{"provingJobId":"1:BASE_PARITY:18c5fa121274369d"}
15:19:24 [15:19:24.818]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:91b6b4bd9a001b4e type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:91b6b4bd9a001b4e"}
15:19:24 [15:19:24.819]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.819]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe3a4699c02dcdb8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe3a4699c02dcdb8"}
15:19:24 [15:19:24.820]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52a673e5b792c1cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:52a673e5b792c1cd"}
15:19:24 [15:19:24.820]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fe3a4699c02dcdb8 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:fe3a4699c02dcdb8"}
15:19:24 [15:19:24.821]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.823]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51bc09e2dca0d874 epochNumber=1
{"provingJobId":"1:BASE_PARITY:51bc09e2dca0d874"}
15:19:24 [15:19:24.823]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:51bc09e2dca0d874
{"provingJobId":"1:BASE_PARITY:51bc09e2dca0d874"}
15:19:24 [15:19:24.823]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:51bc09e2dca0d874 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:51bc09e2dca0d874"}
15:19:24 [15:19:24.824]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4507bc8f681aca80 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4507bc8f681aca80"}
15:19:24 [15:19:24.824]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:51bc09e2dca0d874 has already been completed
{"provingJobId":"1:BASE_PARITY:51bc09e2dca0d874"}
15:19:24 [15:19:24.825]
INFO:
prover-client:proving-broker Proving Broker started
15:19:24 [15:19:24.826]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da39fa641e62a04a epochNumber=1
{"provingJobId":"1:BASE_PARITY:da39fa641e62a04a"}
15:19:24 [15:19:24.826]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:afa4afddb99a5bd2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:afa4afddb99a5bd2"}
15:19:25 [15:19:25.879]
INFO:
prover-client:proving-broker Proving Broker started
15:19:25 [15:19:25.880]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:da39fa641e62a04a settled=false
{"provingJobId":"1:BASE_PARITY:da39fa641e62a04a","status":"pending"}
15:19:25 [15:19:25.880]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:afa4afddb99a5bd2 settled=false
{"provingJobId":"1:BASE_PARITY:afa4afddb99a5bd2","status":"pending"}
15:19:25 [15:19:25.881]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:da39fa641e62a04a type=BASE_PARITY startedAt=1750259964826 older agent has taken job
{"provingJobId":"1:BASE_PARITY:da39fa641e62a04a"}
15:19:25 [15:19:25.881]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:da39fa641e62a04a type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:da39fa641e62a04a"}
15:19:25 [15:19:25.882]
INFO:
prover-client:proving-broker Proving Broker started
15:19:25 [15:19:25.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd20ec99cc10fa24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cd20ec99cc10fa24"}
15:19:25 [15:19:25.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96e624095c3ae85e epochNumber=2
{"provingJobId":"1:BASE_PARITY:96e624095c3ae85e"}
15:19:26 [15:19:26.884]
INFO:
prover-client:proving-broker Proving Broker started
15:19:26 [15:19:26.884]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cd20ec99cc10fa24 settled=false
{"provingJobId":"1:BASE_PARITY:cd20ec99cc10fa24","status":"pending"}
15:19:26 [15:19:26.885]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:96e624095c3ae85e settled=false
{"provingJobId":"1:BASE_PARITY:96e624095c3ae85e","status":"pending"}
15:19:26 [15:19:26.885]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cd20ec99cc10fa24 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:cd20ec99cc10fa24"}
15:19:26 [15:19:26.886]
INFO:
prover-client:proving-broker Proving Broker started
15:19:26 [15:19:26.887]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59225d148a9a4cea epochNumber=1
{"provingJobId":"1:BASE_PARITY:59225d148a9a4cea"}
15:19:26 [15:19:26.887]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed24241ab4f9f62f epochNumber=2
{"provingJobId":"1:BASE_PARITY:ed24241ab4f9f62f"}
15:19:27 [15:19:27.887]
INFO:
prover-client:proving-broker Proving Broker started
15:19:27 [15:19:27.888]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:59225d148a9a4cea settled=false
{"provingJobId":"1:BASE_PARITY:59225d148a9a4cea","status":"pending"}
15:19:27 [15:19:27.888]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ed24241ab4f9f62f settled=false
{"provingJobId":"1:BASE_PARITY:ed24241ab4f9f62f","status":"pending"}
15:19:27 [15:19:27.889]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:59225d148a9a4cea type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:59225d148a9a4cea"}
15:19:27 [15:19:27.890]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:59225d148a9a4cea type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:59225d148a9a4cea"}
15:19:27 [15:19:27.893]
INFO:
prover-client:proving-broker Proving Broker started
15:19:27 [15:19:27.894]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:690745cdec65e549 epochNumber=1
{"provingJobId":"1:BASE_PARITY:690745cdec65e549"}
15:19:27 [15:19:27.894]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:766c9707087b8e83 epochNumber=2
{"provingJobId":"1:BASE_PARITY:766c9707087b8e83"}
15:19:27 [15:19:27.895]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:690745cdec65e549 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:690745cdec65e549"}
15:19:27 [15:19:27.896]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:766c9707087b8e83 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:766c9707087b8e83"}
15:19:27 [15:19:27.897]
INFO:
prover-client:proving-broker Proving Broker started
15:19:27 [15:19:27.898]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e8e504c5cde83c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e8e504c5cde83c6"}
15:19:27 [15:19:27.899]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ae54e62e5ffbcc7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7ae54e62e5ffbcc7"}
15:19:27 [15:19:27.899]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8e8e504c5cde83c6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8e8e504c5cde83c6"}
15:19:27 [15:19:27.900]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8e8e504c5cde83c6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8e8e504c5cde83c6"}
15:19:27 [15:19:27.900]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7ae54e62e5ffbcc7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7ae54e62e5ffbcc7"}
15:19:27 [15:19:27.902]
INFO:
prover-client:proving-broker Proving Broker started
15:19:27 [15:19:27.902]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:21fd8090b93a2a0a err=test error
{"provingJoId":"1:BASE_PARITY:21fd8090b93a2a0a"}
15:19:27 [15:19:27.903]
INFO:
prover-client:proving-broker Proving Broker started
15:19:27 [15:19:27.904]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2ddb5320420154da not found
{"provingJobId":"1:BASE_PARITY:2ddb5320420154da"}
15:19:27 [15:19:27.905]
INFO:
prover-client:proving-broker Proving Broker started
15:19:27 [15:19:27.905]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a854c2c574a8900 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a854c2c574a8900"}
15:19:27 [15:19:27.907]
INFO:
prover-client:proving-broker Proving Broker started
15:19:27 [15:19:27.907]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01d6f402cc841f27 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01d6f402cc841f27"}
15:19:28 [15:19:28.008]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:01d6f402cc841f27 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:01d6f402cc841f27"}
15:19:28 [15:19:28.016]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.018]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbb0da70318238d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dbb0da70318238d7"}
15:19:28 [15:19:28.118]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dbb0da70318238d7 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:dbb0da70318238d7"}
15:19:28 [15:19:28.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c251b6617e1d6fd epochNumber=10
{"provingJobId":"1:BASE_PARITY:6c251b6617e1d6fd"}
15:19:28 [15:19:28.144]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:28 [15:19:28.248]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.249]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04cdde31d20c607f epochNumber=1
{"provingJobId":"1:BASE_PARITY:04cdde31d20c607f"}
15:19:28 [15:19:28.400]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:04cdde31d20c607f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:04cdde31d20c607f"}
15:19:28 [15:19:28.452]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78a4d517b2984891 epochNumber=1
{"provingJobId":"1:BASE_PARITY:78a4d517b2984891"}
15:19:28 [15:19:28.454]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:78a4d517b2984891 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:78a4d517b2984891"}
15:19:28 [15:19:28.455]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.456]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0aa27c14bd6a7e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c0aa27c14bd6a7e2"}
15:19:28 [15:19:28.457]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c0aa27c14bd6a7e2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c0aa27c14bd6a7e2"}
15:19:28 [15:19:28.458]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c0aa27c14bd6a7e2 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:c0aa27c14bd6a7e2"}
15:19:28 [15:19:28.459]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4cfdaa6be44015fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:4cfdaa6be44015fe"}
15:19:28 [15:19:28.460]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4cfdaa6be44015fe type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4cfdaa6be44015fe"}
15:19:28 [15:19:28.461]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.461]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c989420e2f28ca50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c989420e2f28ca50"}
15:19:28 [15:19:28.462]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c989420e2f28ca50 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c989420e2f28ca50"}
15:19:28 [15:19:28.463]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b33f6fbb4dcfb990 epochNumber=3
{"provingJobId":"1:BASE_PARITY:b33f6fbb4dcfb990"}
15:19:28 [15:19:28.486]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:28 [15:19:28.489]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c989420e2f28ca50 err=test error
{"provingJoId":"1:BASE_PARITY:c989420e2f28ca50"}
15:19:28 [15:19:28.490]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.491]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8606974ed2c4f1ff settled=false
{"provingJobId":"1:BASE_PARITY:8606974ed2c4f1ff","status":"pending"}
15:19:28 [15:19:28.491]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:881c59beed366fd9 settled=false
{"provingJobId":"1:BASE_PARITY:881c59beed366fd9","status":"pending"}
15:19:28 [15:19:28.493]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.493]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:26e232958866bc24 settled=true
{"provingJobId":"1:BASE_PARITY:26e232958866bc24","status":"fulfilled"}
15:19:28 [15:19:28.493]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b8a349758548edf9 settled=true
{"provingJobId":"2:BASE_PARITY:b8a349758548edf9","status":"fulfilled"}
15:19:28 [15:19:28.494]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.495]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5601ab8e7506bb53 settled=true
{"provingJobId":"1:BASE_PARITY:5601ab8e7506bb53","status":"fulfilled"}
15:19:28 [15:19:28.495]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6f75027b709d8c62 settled=false
{"provingJobId":"1:BASE_PARITY:6f75027b709d8c62","status":"pending"}
15:19:28 [15:19:28.496]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.497]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d6a5d6e476756d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1d6a5d6e476756d4"}
15:19:28 [15:19:28.499]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1fd738a885d9f32a epochNumber=1
{"provingJobId":"1:BASE_PARITY:1fd738a885d9f32a"}
15:19:28 [15:19:28.501]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:1fd738a885d9f32a: Error: db error: Error: db error
15:19:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:1fd738a885d9f32a"}
15:19:28 [15:19:28.502]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.502]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5cb6eaf7103d4f4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c5cb6eaf7103d4f4"}
15:19:28 [15:19:28.503]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c5cb6eaf7103d4f4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c5cb6eaf7103d4f4"}
15:19:28 [15:19:28.503]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c5cb6eaf7103d4f4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c5cb6eaf7103d4f4"}
15:19:28 [15:19:28.504]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.505]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e643d64af19fe0f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e643d64af19fe0f3"}
15:19:28 [15:19:28.505]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e643d64af19fe0f3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e643d64af19fe0f3"}
15:19:28 [15:19:28.505]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e643d64af19fe0f3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e643d64af19fe0f3"}
15:19:28 [15:19:28.506]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:e643d64af19fe0f3: Error: db error
15:19:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:e643d64af19fe0f3"}
15:19:28 [15:19:28.507]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.508]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:715f93594ad0802a epochNumber=1
{"provingJobId":"1:BASE_PARITY:715f93594ad0802a"}
15:19:28 [15:19:28.508]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:715f93594ad0802a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:715f93594ad0802a"}
15:19:28 [15:19:28.508]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:715f93594ad0802a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:715f93594ad0802a"}
15:19:28 [15:19:28.509]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a656ce5181c976c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a656ce5181c976c6"}
15:19:28 [15:19:28.510]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a656ce5181c976c6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a656ce5181c976c6"}
15:19:28 [15:19:28.511]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a656ce5181c976c6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a656ce5181c976c6"}
15:19:28 [15:19:28.511]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a656ce5181c976c6 jobErr=test error: Error: db error
15:19:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:a656ce5181c976c6"}
15:19:28 [15:19:28.512]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.512]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:615fcc034a9bbf01 not found
{"provingJobId":"1:BASE_PARITY:615fcc034a9bbf01"}
15:19:28 [15:19:28.513]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.514]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:14f7e1787cf7d921 err=test error
{"provingJoId":"1:BASE_PARITY:14f7e1787cf7d921"}
15:19:28 [15:19:28.515]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.525]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5208616ec869c7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:f5208616ec869c7d"}
15:19:28 [15:19:28.526]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f5208616ec869c7d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f5208616ec869c7d"}
15:19:28 [15:19:28.526]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f5208616ec869c7d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f5208616ec869c7d"}
15:19:28 [15:19:28.536]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:e59a27eaff7ea25b epochNumber=2
{"provingJobId":"2:BASE_PARITY:e59a27eaff7ea25b"}
15:19:28 [15:19:28.537]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:e59a27eaff7ea25b type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:e59a27eaff7ea25b"}
15:19:28 [15:19:28.537]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:e59a27eaff7ea25b type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:e59a27eaff7ea25b"}
15:19:28 [15:19:28.599]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:ce3f8826c6602995 epochNumber=3
{"provingJobId":"3:BASE_PARITY:ce3f8826c6602995"}
15:19:28 [15:19:28.617]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:28 [15:19:28.660]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:d402b6b86de4d728 epochNumber=4
{"provingJobId":"4:BASE_PARITY:d402b6b86de4d728"}
15:19:28 [15:19:28.669]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:28 [15:19:28.721]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:eb9fd6ca98c31f86 epochNumber=5
{"provingJobId":"5:BASE_PARITY:eb9fd6ca98c31f86"}
15:19:28 [15:19:28.746]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:28 [15:19:28.777]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:401295a10693ca24 epochNumber=42
{"provingJobId":"1:BASE_PARITY:401295a10693ca24"}
15:19:28 [15:19:28.779]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testQRruPg/42 with map size 1073741824
15:19:28 [15:19:28.784]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:28 [15:19:28.819]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:b81937566bd8ccdb epochNumber=1
{"provingJobId":"1:BASE_PARITY:b81937566bd8ccdb"}
15:19:28 [15:19:28.824]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.825]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e876f9ce684a1e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e876f9ce684a1e5"}
15:19:28 [15:19:28.827]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuVmwbp/1 with map size 1073741824
15:19:28 [15:19:28.831]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:28 [15:19:28.851]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ce3a98c204b1f5f epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ce3a98c204b1f5f"}
15:19:28 [15:19:28.863]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.863]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3103c14d7edc3ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3103c14d7edc3ac"}
15:19:28 [15:19:28.865]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwprwkL/1 with map size 1073741824
15:19:28 [15:19:28.867]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:28 [15:19:28.888]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f3103c14d7edc3ac epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f3103c14d7edc3ac"}
15:19:28 [15:19:28.891]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd118e9b028cf111 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd118e9b028cf111"}
15:19:28 [15:19:28.892]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTXnfAA/1 with map size 1073741824
15:19:28 [15:19:28.894]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:28 [15:19:28.922]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:bd118e9b028cf111 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:bd118e9b028cf111"}
15:19:28 [15:19:28.922]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bd118e9b028cf111 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bd118e9b028cf111"}
15:19:28 [15:19:28.929]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:bd118e9b028cf111 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:bd118e9b028cf111"}
15:19:28 [15:19:28.933]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.934]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:123b59af7cf3f492 epochNumber=1
{"provingJobId":"1:BASE_PARITY:123b59af7cf3f492"}
15:19:28 [15:19:28.935]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCk6IbW/1 with map size 1073741824
15:19:28 [15:19:28.937]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:28 [15:19:28.960]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.962]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d63b49b3a6218e62 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d63b49b3a6218e62"}
15:19:28 [15:19:28.964]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5qrZOB/1 with map size 1073741824
15:19:28 [15:19:28.965]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:28 [15:19:28.983]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d63b49b3a6218e62
{"provingJobId":"1:BASE_PARITY:d63b49b3a6218e62"}
15:19:28 [15:19:28.984]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d63b49b3a6218e62 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d63b49b3a6218e62"}
15:19:28 [15:19:28.984]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d63b49b3a6218e62 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:d63b49b3a6218e62"}
15:19:28 [15:19:28.994]
INFO:
prover-client:proving-broker Proving Broker started
15:19:28 [15:19:28.995]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a06962962a5dcb1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a06962962a5dcb1"}
15:19:28 [15:19:28.995]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaN2d2Z/1 with map size 1073741824
15:19:28 [15:19:28.997]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.021]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1a06962962a5dcb1
{"provingJobId":"1:BASE_PARITY:1a06962962a5dcb1"}
15:19:29 [15:19:29.022]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1a06962962a5dcb1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1a06962962a5dcb1"}
15:19:29 [15:19:29.034]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1626a9e01a57b2d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1626a9e01a57b2d0"}
15:19:29 [15:19:29.037]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9nULkw/1 with map size 1073741824
15:19:29 [15:19:29.043]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.062]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1626a9e01a57b2d0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1626a9e01a57b2d0"}
15:19:29 [15:19:29.063]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1626a9e01a57b2d0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1626a9e01a57b2d0"}
15:19:29 [15:19:29.072]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e70b3f3eb80c22c epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e70b3f3eb80c22c"}
15:19:29 [15:19:29.074]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFYw6fw/1 with map size 1073741824
15:19:29 [15:19:29.079]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.097]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7e70b3f3eb80c22c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7e70b3f3eb80c22c"}
15:19:29 [15:19:29.097]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7e70b3f3eb80c22c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7e70b3f3eb80c22c"}
15:19:29 [15:19:29.109]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.113]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c09a14c812352af epochNumber=0
{"provingJobId":"1:BASE_PARITY:7c09a14c812352af"}
15:19:29 [15:19:29.117]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testXaD1w5/0 with map size 1073741824
15:19:29 [15:19:29.133]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.153]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7c09a14c812352af epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7c09a14c812352af"}
15:19:29 [15:19:29.154]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c9faa6eaf419a0e epochNumber=0
{"provingJobId":"1:BASE_PARITY:8c9faa6eaf419a0e"}
15:19:29 [15:19:29.154]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7c09a14c812352af epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7c09a14c812352af"}
15:19:29 [15:19:29.154]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95e9c8d58f8c7f3a epochNumber=0
{"provingJobId":"1:BASE_PARITY:95e9c8d58f8c7f3a"}
15:19:29 [15:19:29.172]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.175]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.176]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b024442b9d69c3e epochNumber=2
{"provingJobId":"1:BASE_PARITY:6b024442b9d69c3e"}
15:19:29 [15:19:29.177]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZTpRNV/2 with map size 1073741824
15:19:29 [15:19:29.181]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26bfffafaaaf6e1b epochNumber=1
{"provingJobId":"1:BASE_PARITY:26bfffafaaaf6e1b"}
15:19:29 [15:19:29.203]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZTpRNV/1 with map size 1073741824
15:19:29 [15:19:29.206]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.236]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.237]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c86167b553eec3b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c86167b553eec3b1"}
15:19:29 [15:19:29.238]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1e2maj/1 with map size 1073741824
15:19:29 [15:19:29.242]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.265]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d8b635bf1af5d274 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d8b635bf1af5d274"}
15:19:29 [15:19:29.267]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqIUO7X/1 with map size 1073741824
15:19:29 [15:19:29.271]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.294]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb7000b60b9aee3c epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb7000b60b9aee3c"}
15:19:29 [15:19:29.302]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f357ab29ae61f1ef epochNumber=2
{"provingJobId":"1:BASE_PARITY:f357ab29ae61f1ef"}
15:19:29 [15:19:29.303]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqIUO7X/2 with map size 1073741824
15:19:29 [15:19:29.306]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.334]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ecc6cb5bebcf4f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ecc6cb5bebcf4f2"}
15:19:29 [15:19:29.345]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c730f60b9993dba0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c730f60b9993dba0"}
15:19:29 [15:19:29.347]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNd3aOp/1 with map size 1073741824
15:19:29 [15:19:29.349]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80f030a640dc080d epochNumber=1
{"provingJobId":"1:BASE_PARITY:80f030a640dc080d"}
15:19:29 [15:19:29.371]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9bb7fb7714d58a2e epochNumber=2
{"provingJobId":"1:BASE_PARITY:9bb7fb7714d58a2e"}
15:19:29 [15:19:29.372]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNd3aOp/2 with map size 1073741824
15:19:29 [15:19:29.375]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.391]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24c8786c2447d5bc epochNumber=1
{"provingJobId":"1:BASE_PARITY:24c8786c2447d5bc"}
15:19:29 [15:19:29.402]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.402]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ebfe02779bf94e3b epochNumber=1
{"provingJobId":"1:BASE_PARITY:ebfe02779bf94e3b"}
15:19:29 [15:19:29.403]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUJgWBY/1 with map size 1073741824
15:19:29 [15:19:29.406]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.427]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7a2217a4ca9f7ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7a2217a4ca9f7ed"}
15:19:29 [15:19:29.433]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:151114189f7a119d epochNumber=2
{"provingJobId":"1:BASE_PARITY:151114189f7a119d"}
15:19:29 [15:19:29.435]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUJgWBY/2 with map size 1073741824
15:19:29 [15:19:29.441]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.465]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b66e0ba65d0720a epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b66e0ba65d0720a"}
15:19:29 [15:19:29.477]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:919f7a4fcb60137d epochNumber=1
{"provingJobId":"1:BASE_PARITY:919f7a4fcb60137d"}
15:19:29 [15:19:29.479]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVdflGT/1 with map size 1073741824
15:19:29 [15:19:29.483]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.504]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ff4dcb5b3b5138f epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ff4dcb5b3b5138f"}
15:19:29 [15:19:29.516]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:919f7a4fcb60137d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:919f7a4fcb60137d"}
15:19:29 [15:19:29.532]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0db1546f20474795 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0db1546f20474795"}
15:19:29 [15:19:29.535]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuG6jd6/1 with map size 1073741824
15:19:29 [15:19:29.538]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.557]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c07aa56a7586bd4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c07aa56a7586bd4"}
15:19:29 [15:19:29.563]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0db1546f20474795 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:0db1546f20474795"}
15:19:29 [15:19:29.573]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.574]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f02988bfc061745f epochNumber=1
{"provingJobId":"1:BASE_PARITY:f02988bfc061745f"}
15:19:29 [15:19:29.575]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHNoAwu/1 with map size 1073741824
15:19:29 [15:19:29.577]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.595]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6503e59de36b5820 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6503e59de36b5820"}
15:19:29 [15:19:29.600]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f02988bfc061745f type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:f02988bfc061745f"}
15:19:29 [15:19:29.604]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.604]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d83884233f18d5f1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d83884233f18d5f1"}
15:19:29 [15:19:29.605]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3ZVMhi/1 with map size 1073741824
15:19:29 [15:19:29.607]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.632]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d83884233f18d5f1
{"provingJobId":"1:BASE_PARITY:d83884233f18d5f1"}
15:19:29 [15:19:29.633]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d83884233f18d5f1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:d83884233f18d5f1"}
15:19:29 [15:19:29.639]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf07b06215e0e36c epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf07b06215e0e36c"}
15:19:29 [15:19:29.644]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d83884233f18d5f1 has already been completed
{"provingJobId":"1:BASE_PARITY:d83884233f18d5f1"}
15:19:29 [15:19:29.648]
INFO:
prover-client:proving-broker Proving Broker started
15:19:29 [15:19:29.649]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b4f3f225b7bdb4ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:b4f3f225b7bdb4ec"}
15:19:29 [15:19:29.649]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsavcSz/1 with map size 1073741824
15:19:29 [15:19:29.651]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:29 [15:19:29.667]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81e8d2d69fd5993a epochNumber=2
{"provingJobId":"1:BASE_PARITY:81e8d2d69fd5993a"}
15:19:29 [15:19:29.668]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testsavcSz/2 with map size 1073741824
15:19:29 [15:19:29.670]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:30 [15:19:30.737]
INFO:
prover-client:proving-broker Proving Broker started
15:19:30 [15:19:30.743]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b4f3f225b7bdb4ec settled=false
{"provingJobId":"1:BASE_PARITY:b4f3f225b7bdb4ec","status":"pending"}
15:19:30 [15:19:30.745]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:81e8d2d69fd5993a settled=false
{"provingJobId":"1:BASE_PARITY:81e8d2d69fd5993a","status":"pending"}
15:19:30 [15:19:30.746]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b4f3f225b7bdb4ec type=BASE_PARITY startedAt=1750259969685 older agent has taken job
{"provingJobId":"1:BASE_PARITY:b4f3f225b7bdb4ec"}
15:19:30 [15:19:30.746]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b4f3f225b7bdb4ec type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:b4f3f225b7bdb4ec"}
15:19:30 [15:19:30.751]
INFO:
prover-client:proving-broker Proving Broker started
15:19:30 [15:19:30.751]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ca7bae7820300d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ca7bae7820300d7"}
15:19:30 [15:19:30.752]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6an2Ei/1 with map size 1073741824
15:19:30 [15:19:30.756]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:30 [15:19:30.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa8da2eb57ceaf73 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fa8da2eb57ceaf73"}
15:19:30 [15:19:30.773]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6an2Ei/2 with map size 1073741824
15:19:30 [15:19:30.775]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:31 [15:19:31.797]
INFO:
prover-client:proving-broker Proving Broker started
15:19:31 [15:19:31.800]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4ca7bae7820300d7 settled=false
{"provingJobId":"1:BASE_PARITY:4ca7bae7820300d7","status":"pending"}
15:19:31 [15:19:31.802]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fa8da2eb57ceaf73 settled=false
{"provingJobId":"1:BASE_PARITY:fa8da2eb57ceaf73","status":"pending"}
15:19:31 [15:19:31.802]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4ca7bae7820300d7 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:4ca7bae7820300d7"}
15:19:31 [15:19:31.809]
INFO:
prover-client:proving-broker Proving Broker started
15:19:31 [15:19:31.810]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd44d2b50fef305f epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd44d2b50fef305f"}
15:19:31 [15:19:31.817]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTmd37Y/1 with map size 1073741824
15:19:31 [15:19:31.821]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:31 [15:19:31.840]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:608c12ee5ffbc2ba epochNumber=2
{"provingJobId":"1:BASE_PARITY:608c12ee5ffbc2ba"}
15:19:31 [15:19:31.841]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTmd37Y/2 with map size 1073741824
15:19:31 [15:19:31.844]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:32 [15:19:32.863]
INFO:
prover-client:proving-broker Proving Broker started
15:19:32 [15:19:32.865]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dd44d2b50fef305f settled=false
{"provingJobId":"1:BASE_PARITY:dd44d2b50fef305f","status":"pending"}
15:19:32 [15:19:32.867]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:608c12ee5ffbc2ba settled=false
{"provingJobId":"1:BASE_PARITY:608c12ee5ffbc2ba","status":"pending"}
15:19:32 [15:19:32.868]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dd44d2b50fef305f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dd44d2b50fef305f"}
15:19:32 [15:19:32.868]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dd44d2b50fef305f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dd44d2b50fef305f"}
15:19:32 [15:19:32.878]
INFO:
prover-client:proving-broker Proving Broker started
15:19:32 [15:19:32.879]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f461a88d82eff16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f461a88d82eff16"}
15:19:32 [15:19:32.879]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcECFiv/1 with map size 1073741824
15:19:32 [15:19:32.881]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:32 [15:19:32.899]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43ce2b3d8214ec15 epochNumber=2
{"provingJobId":"1:BASE_PARITY:43ce2b3d8214ec15"}
15:19:32 [15:19:32.900]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcECFiv/2 with map size 1073741824
15:19:32 [15:19:32.902]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:32 [15:19:32.923]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5f461a88d82eff16 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5f461a88d82eff16"}
15:19:32 [15:19:32.930]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:43ce2b3d8214ec15 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:43ce2b3d8214ec15"}
15:19:32 [15:19:32.941]
INFO:
prover-client:proving-broker Proving Broker started
15:19:32 [15:19:32.942]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c8ee35adf2e886a epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c8ee35adf2e886a"}
15:19:32 [15:19:32.944]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfhxs7O/1 with map size 1073741824
15:19:32 [15:19:32.947]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:32 [15:19:32.969]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3945fd1254a24ae6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3945fd1254a24ae6"}
15:19:32 [15:19:32.970]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfhxs7O/2 with map size 1073741824
15:19:32 [15:19:32.974]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:32 [15:19:32.991]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3c8ee35adf2e886a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3c8ee35adf2e886a"}
15:19:32 [15:19:32.992]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3c8ee35adf2e886a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3c8ee35adf2e886a"}
15:19:32 [15:19:32.998]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3945fd1254a24ae6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3945fd1254a24ae6"}
15:19:33 [15:19:33.008]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.011]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bba60477fefd0faf err=test error
{"provingJoId":"1:BASE_PARITY:bba60477fefd0faf"}
15:19:33 [15:19:33.021]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.022]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f3c0a305222d765c not found
{"provingJobId":"1:BASE_PARITY:f3c0a305222d765c"}
15:19:33 [15:19:33.029]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.030]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d28c343269b82287 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d28c343269b82287"}
15:19:33 [15:19:33.031]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSQE3fG/1 with map size 1073741824
15:19:33 [15:19:33.036]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.061]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.062]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b9b2c1cac66a488 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b9b2c1cac66a488"}
15:19:33 [15:19:33.064]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvdDzRY/1 with map size 1073741824
15:19:33 [15:19:33.068]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.214]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6b9b2c1cac66a488 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6b9b2c1cac66a488"}
15:19:33 [15:19:33.228]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.229]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4112bfc9a6841f6f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4112bfc9a6841f6f"}
15:19:33 [15:19:33.230]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAO8B4H/1 with map size 1073741824
15:19:33 [15:19:33.233]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.358]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4112bfc9a6841f6f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4112bfc9a6841f6f"}
15:19:33 [15:19:33.361]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4923153e73927175 epochNumber=10
{"provingJobId":"1:BASE_PARITY:4923153e73927175"}
15:19:33 [15:19:33.363]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testAO8B4H/10 with map size 1073741824
15:19:33 [15:19:33.366]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.409]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:33 [15:19:33.410]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:19:33 [15:19:33.517]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.518]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b921b88cb8702f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b921b88cb8702f3"}
15:19:33 [15:19:33.518]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzi7XFf/1 with map size 1073741824
15:19:33 [15:19:33.520]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.721]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9b921b88cb8702f3 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9b921b88cb8702f3"}
15:19:33 [15:19:33.759]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.760]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c74ebf1d3354f105 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c74ebf1d3354f105"}
15:19:33 [15:19:33.761]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkDUuxi/1 with map size 1073741824
15:19:33 [15:19:33.765]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.786]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c74ebf1d3354f105 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c74ebf1d3354f105"}
15:19:33 [15:19:33.791]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.792]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25a5351caaedc050 epochNumber=1
{"provingJobId":"1:BASE_PARITY:25a5351caaedc050"}
15:19:33 [15:19:33.793]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3ahNBL/1 with map size 1073741824
15:19:33 [15:19:33.795]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.824]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:25a5351caaedc050 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:25a5351caaedc050"}
15:19:33 [15:19:33.826]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:25a5351caaedc050 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:25a5351caaedc050"}
15:19:33 [15:19:33.837]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.838]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46a337cf5255dd05 epochNumber=1
{"provingJobId":"1:BASE_PARITY:46a337cf5255dd05"}
15:19:33 [15:19:33.839]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testS7N7WL/1 with map size 1073741824
15:19:33 [15:19:33.841]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.866]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:46a337cf5255dd05 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:46a337cf5255dd05"}
15:19:33 [15:19:33.878]
INFO:
prover-client:proving-broker Proving Broker started
15:19:33 [15:19:33.879]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90c60f34d64b85f4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:90c60f34d64b85f4"}
15:19:33 [15:19:33.880]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testarEPLg/1 with map size 1073741824
15:19:33 [15:19:33.883]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.907]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:90c60f34d64b85f4 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:90c60f34d64b85f4"}
15:19:33 [15:19:33.908]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a459ef02db5141b epochNumber=3
{"provingJobId":"1:BASE_PARITY:9a459ef02db5141b"}
15:19:33 [15:19:33.909]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testarEPLg/3 with map size 1073741824
15:19:33 [15:19:33.913]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.955]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:33 [15:19:33.955]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:19:33 [15:19:33.963]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:90c60f34d64b85f4 err=test error
{"provingJoId":"1:BASE_PARITY:90c60f34d64b85f4"}
15:19:33 [15:19:33.971]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCjCCX0/1 with map size 1073741824
15:19:33 [15:19:33.974]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:33 [15:19:33.998]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCjCCX0/2 with map size 1073741824
15:19:34 [15:19:34.002]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.019]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.021]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f35b1f3f5cb0fcc1 settled=false
{"provingJobId":"1:BASE_PARITY:f35b1f3f5cb0fcc1","status":"pending"}
15:19:34 [15:19:34.022]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:790c251e5c107511 settled=false
{"provingJobId":"1:BASE_PARITY:790c251e5c107511","status":"pending"}
15:19:34 [15:19:34.027]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIeBLkS/1 with map size 1073741824
15:19:34 [15:19:34.028]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.052]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIeBLkS/2 with map size 1073741824
15:19:34 [15:19:34.057]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.093]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.097]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fd1b854542df0d5b settled=true
{"provingJobId":"1:BASE_PARITY:fd1b854542df0d5b","status":"fulfilled"}
15:19:34 [15:19:34.099]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:8107cdf29a4ca354 settled=true
{"provingJobId":"2:BASE_PARITY:8107cdf29a4ca354","status":"fulfilled"}
15:19:34 [15:19:34.106]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIOHNMq/1 with map size 1073741824
15:19:34 [15:19:34.109]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.137]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIOHNMq/2 with map size 1073741824
15:19:34 [15:19:34.140]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.164]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.165]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:96cbf642ab41efc0 settled=true
{"provingJobId":"1:BASE_PARITY:96cbf642ab41efc0","status":"fulfilled"}
15:19:34 [15:19:34.167]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1954679dbbd5ae8e settled=false
{"provingJobId":"1:BASE_PARITY:1954679dbbd5ae8e","status":"pending"}
15:19:34 [15:19:34.175]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.175]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c642a51f99df050 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c642a51f99df050"}
15:19:34 [15:19:34.176]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcIv9LF/1 with map size 1073741824
15:19:34 [15:19:34.180]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.207]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc61c2d6ade3dfd6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc61c2d6ade3dfd6"}
15:19:34 [15:19:34.209]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:fc61c2d6ade3dfd6: Error: db error: Error: db error
15:19:34 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:fc61c2d6ade3dfd6"}
15:19:34 [15:19:34.216]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.217]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87655209485aab76 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87655209485aab76"}
15:19:34 [15:19:34.222]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfPbwXr/1 with map size 1073741824
15:19:34 [15:19:34.224]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.244]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:87655209485aab76 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:87655209485aab76"}
15:19:34 [15:19:34.244]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:87655209485aab76 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:87655209485aab76"}
15:19:34 [15:19:34.255]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7b070019b10d879 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7b070019b10d879"}
15:19:34 [15:19:34.256]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNMlutZ/1 with map size 1073741824
15:19:34 [15:19:34.258]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.280]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b7b070019b10d879 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b7b070019b10d879"}
15:19:34 [15:19:34.281]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b7b070019b10d879 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b7b070019b10d879"}
15:19:34 [15:19:34.282]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:b7b070019b10d879: Error: db error
15:19:34 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:b7b070019b10d879"}
15:19:34 [15:19:34.285]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3358f051d243d0ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:3358f051d243d0ed"}
15:19:34 [15:19:34.286]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdqzoPI/1 with map size 1073741824
15:19:34 [15:19:34.288]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.312]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3358f051d243d0ed type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3358f051d243d0ed"}
15:19:34 [15:19:34.312]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3358f051d243d0ed type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3358f051d243d0ed"}
15:19:34 [15:19:34.322]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.323]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f560653ad694619 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f560653ad694619"}
15:19:34 [15:19:34.324]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPSvPiu/1 with map size 1073741824
15:19:34 [15:19:34.326]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.343]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4f560653ad694619 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4f560653ad694619"}
15:19:34 [15:19:34.343]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4f560653ad694619 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4f560653ad694619"}
15:19:34 [15:19:34.344]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:4f560653ad694619 jobErr=test error: Error: db error
15:19:34 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:4f560653ad694619"}
15:19:34 [15:19:34.350]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.351]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:071891c181ea79cd not found
{"provingJobId":"1:BASE_PARITY:071891c181ea79cd"}
15:19:34 [15:19:34.356]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.356]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2e2c6f26e24ac6e1 err=test error
{"provingJoId":"1:BASE_PARITY:2e2c6f26e24ac6e1"}
15:19:34 [15:19:34.362]
INFO:
prover-client:proving-broker Proving Broker started
15:19:34 [15:19:34.372]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:155faf7a89465e57 epochNumber=1
{"provingJobId":"1:BASE_PARITY:155faf7a89465e57"}
15:19:34 [15:19:34.374]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlHxVn4/1 with map size 1073741824
15:19:34 [15:19:34.375]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.393]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:155faf7a89465e57 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:155faf7a89465e57"}
15:19:34 [15:19:34.394]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:155faf7a89465e57 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:155faf7a89465e57"}
15:19:34 [15:19:34.409]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:63bd95e2617914a7 epochNumber=2
{"provingJobId":"2:BASE_PARITY:63bd95e2617914a7"}
15:19:34 [15:19:34.410]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlHxVn4/2 with map size 1073741824
15:19:34 [15:19:34.412]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.430]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:63bd95e2617914a7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:63bd95e2617914a7"}
15:19:34 [15:19:34.430]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:63bd95e2617914a7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:63bd95e2617914a7"}
15:19:34 [15:19:34.496]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:2df0409a989f828a epochNumber=3
{"provingJobId":"3:BASE_PARITY:2df0409a989f828a"}
15:19:34 [15:19:34.497]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testlHxVn4/3 with map size 1073741824
15:19:34 [15:19:34.499]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.542]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:34 [15:19:34.543]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:19:34 [15:19:34.582]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:0b55b4e2dac68eaf epochNumber=4
{"provingJobId":"4:BASE_PARITY:0b55b4e2dac68eaf"}
15:19:34 [15:19:34.583]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testlHxVn4/4 with map size 1073741824
15:19:34 [15:19:34.584]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.623]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:34 [15:19:34.624]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
15:19:34 [15:19:34.667]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:25fc5a6e803ce347 epochNumber=5
{"provingJobId":"5:BASE_PARITY:25fc5a6e803ce347"}
15:19:34 [15:19:34.668]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testlHxVn4/5 with map size 1073741824
15:19:34 [15:19:34.673]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:19:34 [15:19:34.709]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:19:34 [15:19:34.710]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
15:19:34
PASS src/proving_broker/proving_broker.test.ts (
13.594 s)
15:19:34 ProvingBroker
15:19:34 Producer API
15:19:34
✓ refuses stale jobs (50 ms)
15:19:34
✓ enqueues jobs (3 ms)
15:19:34
✓ ignores duplicate jobs (3 ms)
15:19:34
✓ reports correct status when enqueuing repeat jobs (4 ms)
15:19:34
✓ throws an error in case of duplicate job IDs (26 ms)
15:19:34
✓ returns not-found status for non-existing jobs (1 ms)
15:19:34
✓ cancels jobs in queue (2 ms)
15:19:34
✓ cancels jobs in-progress (2 ms)
15:19:34
✓ returns job result if successful (2 ms)
15:19:34
✓ returns job error if failed (2 ms)
15:19:34
✓ correctly returns job status for concurrent writes (3 ms)
15:19:34
✓ refuses stale jobs (48 ms)
15:19:34
✓ enqueues jobs (37 ms)
15:19:34
✓ ignores duplicate jobs (29 ms)
15:19:34
✓ reports correct status when enqueuing repeat jobs (40 ms)
15:19:34
✓ throws an error in case of duplicate job IDs (26 ms)
15:19:34
✓ returns not-found status for non-existing jobs (3 ms)
15:19:34
✓ cancels jobs in queue (32 ms)
15:19:34
✓ cancels jobs in-progress (39 ms)
15:19:34
✓ returns job result if successful (37 ms)
15:19:34
✓ returns job error if failed (35 ms)
15:19:34
✓ correctly returns job status for concurrent writes (65 ms)
15:19:34 Consumer API
15:19:34
✓ returns undefined if no jobs are available (8 ms)
15:19:34
✓ returns jobs in priority order (2 ms)
15:19:34
✓ returns undefined if no jobs are available for the given allowList (1 ms)
15:19:34
✓ returns a job if it is in the allowList (2 ms)
15:19:34
✓ returns the most important job if it is in the allowList (1 ms)
15:19:34
✓ returns any job if filter is empty (2 ms)
15:19:34
✓ returns a new job when reporting job success (6 ms)
15:19:34
✓ returns a new job when reporting permanent error (2 ms)
15:19:34
✓ returns a new job when reporting retry-able error (2 ms)
15:19:34
✓ returns a new job when reporting progress if current one is cancelled (4 ms)
15:19:34
✓ returns a new job if job is already in progress elsewhere (1057 ms)
15:19:34
✓ avoids sending the same job to a new agent after a restart (1004 ms)
15:19:34
✓ avoids sending a completed job to a new agent after a restart (1004 ms)
15:19:34
✓ tracks job result if in progress (5 ms)
15:19:34
✓ tracks job result even if job is in queue (4 ms)
15:19:34
✓ ignores reported job error if unknown job (2 ms)
15:19:34
✓ ignores job result if unknown job (1 ms)
15:19:34
✓ returns undefined if no jobs are available (2 ms)
15:19:34
✓ returns jobs in priority order (59 ms)
15:19:34
✓ returns undefined if no jobs are available for the given allowList (30 ms)
15:19:34
✓ returns a job if it is in the allowList (79 ms)
15:19:34
✓ returns the most important job if it is in the allowList (58 ms)
15:19:34
✓ returns any job if filter is empty (73 ms)
15:19:34
✓ returns a new job when reporting job success (55 ms)
15:19:34
✓ returns a new job when reporting permanent error (43 ms)
15:19:34
✓ returns a new job when reporting retry-able error (31 ms)
15:19:34
✓ returns a new job when reporting progress if current one is cancelled (44 ms)
15:19:34
✓ returns a new job if job is already in progress elsewhere (1102 ms)
15:19:34
✓ avoids sending the same job to a new agent after a restart (1057 ms)
15:19:34
✓ avoids sending a completed job to a new agent after a restart (1069 ms)
15:19:34
✓ tracks job result if in progress (62 ms)
15:19:34
✓ tracks job result even if job is in queue (67 ms)
15:19:34
✓ ignores reported job error if unknown job (6 ms)
15:19:34
✓ ignores job result if unknown job (11 ms)
15:19:34 Timeouts
15:19:34
✓ tracks in progress jobs (1 ms)
15:19:34
✓ re-enqueues jobs that time out (109 ms)
15:19:34
✓ cancel stale jobs that time out (232 ms)
15:19:34
✓ keeps the jobs in progress while it is alive (203 ms)
15:19:34
✓ tracks in progress jobs (35 ms)
15:19:34
✓ re-enqueues jobs that time out (161 ms)
15:19:34
✓ cancel stale jobs that time out (294 ms)
15:19:34
✓ keeps the jobs in progress while it is alive (237 ms)
15:19:34 Retries
15:19:34
✓ retries jobs (3 ms)
15:19:34
✓ retries up to a maximum number of times (3 ms)
15:19:34
✓ passing retry=false does not retry (2 ms)
15:19:34
✓ does not retry if job is stale (29 ms)
15:19:34
✓ retries jobs (36 ms)
15:19:34
✓ retries up to a maximum number of times (45 ms)
15:19:34
✓ passing retry=false does not retry (40 ms)
15:19:34
✓ does not retry if job is stale (88 ms)
15:19:34 Database management
15:19:34
✓ re-enqueues proof requests on start (2 ms)
15:19:34
✓ restores proof results on start (2 ms)
15:19:34
✓ only re-enqueues unfinished jobs (1 ms)
15:19:34
✓ saves job when enqueued (3 ms)
15:19:34
✓ does not retain job if database fails to save (2 ms)
15:19:34
✓ saves job result (2 ms)
15:19:34
✓ saves result even if database fails to save (2 ms)
15:19:34
✓ saves job error (2 ms)
15:19:34
✓ saves job error even if database fails to save (3 ms)
15:19:34
✓ does not save job result if job is unknown (1 ms)
15:19:34
✓ does not save job error if job is unknown (1 ms)
15:19:34
✓ cleans up old jobs periodically (260 ms)
15:19:34
✓ re-enqueues proof requests on start (59 ms)
15:19:34
✓ restores proof results on start (78 ms)
15:19:34
✓ only re-enqueues unfinished jobs (67 ms)
15:19:34
✓ saves job when enqueued (31 ms)
15:19:34
✓ does not retain job if database fails to save (9 ms)
15:19:34
✓ saves job result (39 ms)
15:19:34
✓ saves result even if database fails to save (31 ms)
15:19:34
✓ saves job error (37 ms)
15:19:34
✓ saves job error even if database fails to save (28 ms)
15:19:34
✓ does not save job result if job is unknown (4 ms)
15:19:34
✓ does not save job error if job is unknown (5 ms)
15:19:34
✓ cleans up old jobs periodically (389 ms)
15:19:34
15:19:34
Test Suites: 1 passed, 1 total
15:19:34
Tests: 96 passed, 96 total
15:19:34
Snapshots: 0 total
15:19:34
Time: 13.694 s
15:19:34
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
15:19:34
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?