Parent Log:
http://ci.aztec-labs.com/fc0133061c7951d2
Command: abf2df8792edab68 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/15da5b0f4fc17ee864697604093cf1ff2bd900d2
Env: REF_NAME=gh-readonly-queue/staging/pr-15174-a1fcc6d25d2ef5242bc875b44f6b8c627357a0d1 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 20 12:51:28 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15174_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_staging
12:51:32 [12:51:32.667]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.673]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96a5ae6c0fee9a94 epochNumber=42
{"provingJobId":"1:BASE_PARITY:96a5ae6c0fee9a94"}
12:51:32 [12:51:32.676]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:667535c285c9a01c epochNumber=1
{"provingJobId":"1:BASE_PARITY:667535c285c9a01c"}
12:51:32 [12:51:32.708]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.709]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5822b57a393497d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5822b57a393497d6"}
12:51:32 [12:51:32.710]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f701d7b56def8d23 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f701d7b56def8d23"}
12:51:32 [12:51:32.711]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.712]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ca7a17ee8e5f598 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ca7a17ee8e5f598"}
12:51:32 [12:51:32.713]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5ca7a17ee8e5f598 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5ca7a17ee8e5f598"}
12:51:32 [12:51:32.715]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.716]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:611729880d5e79c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:611729880d5e79c5"}
12:51:32 [12:51:32.717]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:611729880d5e79c5 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:611729880d5e79c5"}
12:51:32 [12:51:32.717]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:611729880d5e79c5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:611729880d5e79c5"}
12:51:32 [12:51:32.718]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:611729880d5e79c5 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:611729880d5e79c5"}
12:51:32 [12:51:32.719]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.720]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2eafb1bda016989 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e2eafb1bda016989"}
12:51:32 [12:51:32.745]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.747]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.747]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee25c9282598273a epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee25c9282598273a"}
12:51:32 [12:51:32.748]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ee25c9282598273a
{"provingJobId":"1:BASE_PARITY:ee25c9282598273a"}
12:51:32 [12:51:32.748]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ee25c9282598273a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ee25c9282598273a"}
12:51:32 [12:51:32.749]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ee25c9282598273a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ee25c9282598273a"}
12:51:32 [12:51:32.750]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.751]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41dec7bb091a99cf epochNumber=1
{"provingJobId":"1:BASE_PARITY:41dec7bb091a99cf"}
12:51:32 [12:51:32.751]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:41dec7bb091a99cf
{"provingJobId":"1:BASE_PARITY:41dec7bb091a99cf"}
12:51:32 [12:51:32.752]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:41dec7bb091a99cf type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:41dec7bb091a99cf"}
12:51:32 [12:51:32.753]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.754]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66d3a096f47c3535 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66d3a096f47c3535"}
12:51:32 [12:51:32.754]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:66d3a096f47c3535 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:66d3a096f47c3535"}
12:51:32 [12:51:32.755]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:66d3a096f47c3535 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:66d3a096f47c3535"}
12:51:32 [12:51:32.756]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.757]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0eb7136a8192465c epochNumber=1
{"provingJobId":"1:BASE_PARITY:0eb7136a8192465c"}
12:51:32 [12:51:32.757]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0eb7136a8192465c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0eb7136a8192465c"}
12:51:32 [12:51:32.757]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0eb7136a8192465c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0eb7136a8192465c"}
12:51:32 [12:51:32.758]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.759]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dcb6708e663da1bd epochNumber=0
{"provingJobId":"1:BASE_PARITY:dcb6708e663da1bd"}
12:51:32 [12:51:32.759]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dcb6708e663da1bd epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dcb6708e663da1bd"}
12:51:32 [12:51:32.760]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7837a771bb3be4aa epochNumber=0
{"provingJobId":"1:BASE_PARITY:7837a771bb3be4aa"}
12:51:32 [12:51:32.760]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dcb6708e663da1bd epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dcb6708e663da1bd"}
12:51:32 [12:51:32.760]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c62d889ff97768a epochNumber=0
{"provingJobId":"1:BASE_PARITY:7c62d889ff97768a"}
12:51:32 [12:51:32.770]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.771]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:690d9bc5f8ab67bc epochNumber=2
{"provingJobId":"1:BASE_PARITY:690d9bc5f8ab67bc"}
12:51:32 [12:51:32.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d04645ac36e80009 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d04645ac36e80009"}
12:51:32 [12:51:32.774]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.774]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a005dd025f69c0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a005dd025f69c0f"}
12:51:32 [12:51:32.776]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2281786ff257709e epochNumber=1
{"provingJobId":"1:BASE_PARITY:2281786ff257709e"}
12:51:32 [12:51:32.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4e32e81bdfa9040 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4e32e81bdfa9040"}
12:51:32 [12:51:32.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de7b17067f89226a epochNumber=2
{"provingJobId":"1:BASE_PARITY:de7b17067f89226a"}
12:51:32 [12:51:32.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81165638d2d234a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:81165638d2d234a8"}
12:51:32 [12:51:32.779]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.780]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ef853a21683b0eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ef853a21683b0eb"}
12:51:32 [12:51:32.781]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a663e2de985abe41 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a663e2de985abe41"}
12:51:32 [12:51:32.781]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f1c2088d058582df epochNumber=2
{"provingJobId":"1:BASE_PARITY:f1c2088d058582df"}
12:51:32 [12:51:32.781]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c750ef9d23788f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c750ef9d23788f2"}
12:51:32 [12:51:32.782]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:afd74d640f91fcab epochNumber=1
{"provingJobId":"1:BASE_PARITY:afd74d640f91fcab"}
12:51:32 [12:51:32.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:db84197b385edb42 epochNumber=1
{"provingJobId":"1:BASE_PARITY:db84197b385edb42"}
12:51:32 [12:51:32.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04f80f77cc6c40e0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:04f80f77cc6c40e0"}
12:51:32 [12:51:32.784]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:510b9d91815a30b2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:510b9d91815a30b2"}
12:51:32 [12:51:32.785]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28702a4a1b180fd8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:28702a4a1b180fd8"}
12:51:32 [12:51:32.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32215a287aa02a87 epochNumber=1
{"provingJobId":"1:BASE_PARITY:32215a287aa02a87"}
12:51:32 [12:51:32.786]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:28702a4a1b180fd8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:28702a4a1b180fd8"}
12:51:32 [12:51:32.787]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.788]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b069f8a5f4742c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b069f8a5f4742c3"}
12:51:32 [12:51:32.788]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11f001b82c3d3883 epochNumber=1
{"provingJobId":"1:BASE_PARITY:11f001b82c3d3883"}
12:51:32 [12:51:32.788]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4b069f8a5f4742c3 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:4b069f8a5f4742c3"}
12:51:32 [12:51:32.789]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.790]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3dcfdde5df84b43e epochNumber=1
{"provingJobId":"1:BASE_PARITY:3dcfdde5df84b43e"}
12:51:32 [12:51:32.790]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c6c942edaeaa417 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c6c942edaeaa417"}
12:51:32 [12:51:32.790]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3dcfdde5df84b43e type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:3dcfdde5df84b43e"}
12:51:32 [12:51:32.791]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.792]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6daf4db7a5555074 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6daf4db7a5555074"}
12:51:32 [12:51:32.792]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6daf4db7a5555074
{"provingJobId":"1:BASE_PARITY:6daf4db7a5555074"}
12:51:32 [12:51:32.792]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6daf4db7a5555074 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:6daf4db7a5555074"}
12:51:32 [12:51:32.793]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0be656387fd37b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0be656387fd37b0"}
12:51:32 [12:51:32.793]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6daf4db7a5555074 has already been completed
{"provingJobId":"1:BASE_PARITY:6daf4db7a5555074"}
12:51:32 [12:51:32.794]
INFO:
prover-client:proving-broker Proving Broker started
12:51:32 [12:51:32.795]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:524933a2a51c3268 epochNumber=1
{"provingJobId":"1:BASE_PARITY:524933a2a51c3268"}
12:51:32 [12:51:32.795]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:49aa64c1c3709231 epochNumber=2
{"provingJobId":"1:BASE_PARITY:49aa64c1c3709231"}
12:51:33 [12:51:33.848]
INFO:
prover-client:proving-broker Proving Broker started
12:51:33 [12:51:33.849]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:524933a2a51c3268 settled=false
{"provingJobId":"1:BASE_PARITY:524933a2a51c3268","status":"pending"}
12:51:33 [12:51:33.850]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:49aa64c1c3709231 settled=false
{"provingJobId":"1:BASE_PARITY:49aa64c1c3709231","status":"pending"}
12:51:33 [12:51:33.851]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:524933a2a51c3268 type=BASE_PARITY startedAt=1750423892795 older agent has taken job
{"provingJobId":"1:BASE_PARITY:524933a2a51c3268"}
12:51:33 [12:51:33.852]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:524933a2a51c3268 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:524933a2a51c3268"}
12:51:33 [12:51:33.853]
INFO:
prover-client:proving-broker Proving Broker started
12:51:33 [12:51:33.854]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5998dcf7bacb764d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5998dcf7bacb764d"}
12:51:33 [12:51:33.854]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53d09865ef4073e1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:53d09865ef4073e1"}
12:51:34 [12:51:34.856]
INFO:
prover-client:proving-broker Proving Broker started
12:51:34 [12:51:34.857]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5998dcf7bacb764d settled=false
{"provingJobId":"1:BASE_PARITY:5998dcf7bacb764d","status":"pending"}
12:51:34 [12:51:34.858]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:53d09865ef4073e1 settled=false
{"provingJobId":"1:BASE_PARITY:53d09865ef4073e1","status":"pending"}
12:51:34 [12:51:34.859]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5998dcf7bacb764d type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:5998dcf7bacb764d"}
12:51:34 [12:51:34.861]
INFO:
prover-client:proving-broker Proving Broker started
12:51:34 [12:51:34.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c33cad8acce429f6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c33cad8acce429f6"}
12:51:34 [12:51:34.863]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee7a8d7c45943ab1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ee7a8d7c45943ab1"}
12:51:35 [12:51:35.863]
INFO:
prover-client:proving-broker Proving Broker started
12:51:35 [12:51:35.864]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c33cad8acce429f6 settled=false
{"provingJobId":"1:BASE_PARITY:c33cad8acce429f6","status":"pending"}
12:51:35 [12:51:35.864]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ee7a8d7c45943ab1 settled=false
{"provingJobId":"1:BASE_PARITY:ee7a8d7c45943ab1","status":"pending"}
12:51:35 [12:51:35.865]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c33cad8acce429f6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c33cad8acce429f6"}
12:51:35 [12:51:35.865]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c33cad8acce429f6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c33cad8acce429f6"}
12:51:35 [12:51:35.866]
INFO:
prover-client:proving-broker Proving Broker started
12:51:35 [12:51:35.867]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:556e5ec0681cc6fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:556e5ec0681cc6fb"}
12:51:35 [12:51:35.867]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3154e7841f2151b epochNumber=2
{"provingJobId":"1:BASE_PARITY:a3154e7841f2151b"}
12:51:35 [12:51:35.867]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:556e5ec0681cc6fb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:556e5ec0681cc6fb"}
12:51:35 [12:51:35.868]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a3154e7841f2151b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a3154e7841f2151b"}
12:51:35 [12:51:35.868]
INFO:
prover-client:proving-broker Proving Broker started
12:51:35 [12:51:35.869]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80c8a9b5d87f68a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:80c8a9b5d87f68a8"}
12:51:35 [12:51:35.869]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31e15abb7e2168f9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:31e15abb7e2168f9"}
12:51:35 [12:51:35.869]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:80c8a9b5d87f68a8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:80c8a9b5d87f68a8"}
12:51:35 [12:51:35.869]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:80c8a9b5d87f68a8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:80c8a9b5d87f68a8"}
12:51:35 [12:51:35.870]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:31e15abb7e2168f9 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:31e15abb7e2168f9"}
12:51:35 [12:51:35.870]
INFO:
prover-client:proving-broker Proving Broker started
12:51:35 [12:51:35.871]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d1d3aa4721f3cf22 err=test error
{"provingJoId":"1:BASE_PARITY:d1d3aa4721f3cf22"}
12:51:35 [12:51:35.871]
INFO:
prover-client:proving-broker Proving Broker started
12:51:35 [12:51:35.872]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7d22f88c61f970c6 not found
{"provingJobId":"1:BASE_PARITY:7d22f88c61f970c6"}
12:51:35 [12:51:35.873]
INFO:
prover-client:proving-broker Proving Broker started
12:51:35 [12:51:35.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eafaba2c35b3b7a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eafaba2c35b3b7a0"}
12:51:35 [12:51:35.874]
INFO:
prover-client:proving-broker Proving Broker started
12:51:35 [12:51:35.874]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5c35e2b696c1ca7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5c35e2b696c1ca7"}
12:51:35 [12:51:35.976]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e5c35e2b696c1ca7 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e5c35e2b696c1ca7"}
12:51:35 [12:51:35.983]
INFO:
prover-client:proving-broker Proving Broker started
12:51:35 [12:51:35.984]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64eabeed75c18e92 epochNumber=1
{"provingJobId":"1:BASE_PARITY:64eabeed75c18e92"}
12:51:36 [12:51:36.086]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:64eabeed75c18e92 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:64eabeed75c18e92"}
12:51:36 [12:51:36.092]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e48e0c506a34193 epochNumber=10
{"provingJobId":"1:BASE_PARITY:7e48e0c506a34193"}
12:51:36 [12:51:36.113]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:36 [12:51:36.219]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.220]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e0c8ab021164987 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e0c8ab021164987"}
12:51:36 [12:51:36.373]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6e0c8ab021164987 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6e0c8ab021164987"}
12:51:36 [12:51:36.425]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.426]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f06a7468af399373 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f06a7468af399373"}
12:51:36 [12:51:36.427]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f06a7468af399373 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f06a7468af399373"}
12:51:36 [12:51:36.428]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.429]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4157f3c709b498b epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4157f3c709b498b"}
12:51:36 [12:51:36.429]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c4157f3c709b498b type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c4157f3c709b498b"}
12:51:36 [12:51:36.430]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c4157f3c709b498b type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:c4157f3c709b498b"}
12:51:36 [12:51:36.431]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.432]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f4a227798eda682 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f4a227798eda682"}
12:51:36 [12:51:36.432]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3f4a227798eda682 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3f4a227798eda682"}
12:51:36 [12:51:36.433]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.434]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae67a31cb55af196 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae67a31cb55af196"}
12:51:36 [12:51:36.434]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ae67a31cb55af196 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:ae67a31cb55af196"}
12:51:36 [12:51:36.435]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7adf6e413514a6e epochNumber=3
{"provingJobId":"1:BASE_PARITY:e7adf6e413514a6e"}
12:51:36 [12:51:36.460]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:36 [12:51:36.460]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ae67a31cb55af196 err=test error
{"provingJoId":"1:BASE_PARITY:ae67a31cb55af196"}
12:51:36 [12:51:36.462]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.462]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1fffe47225e5250e settled=false
{"provingJobId":"1:BASE_PARITY:1fffe47225e5250e","status":"pending"}
12:51:36 [12:51:36.463]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ccb127961082f40d settled=false
{"provingJobId":"1:BASE_PARITY:ccb127961082f40d","status":"pending"}
12:51:36 [12:51:36.465]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.465]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7b134e79ddabb91e settled=true
{"provingJobId":"1:BASE_PARITY:7b134e79ddabb91e","status":"fulfilled"}
12:51:36 [12:51:36.465]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:7c4013e6082b96de settled=true
{"provingJobId":"2:BASE_PARITY:7c4013e6082b96de","status":"fulfilled"}
12:51:36 [12:51:36.466]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.467]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:99c9e349953ade0f settled=true
{"provingJobId":"1:BASE_PARITY:99c9e349953ade0f","status":"fulfilled"}
12:51:36 [12:51:36.467]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b96d61e484815d8e settled=false
{"provingJobId":"1:BASE_PARITY:b96d61e484815d8e","status":"pending"}
12:51:36 [12:51:36.468]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.469]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3462b18214c3327d epochNumber=1
{"provingJobId":"1:BASE_PARITY:3462b18214c3327d"}
12:51:36 [12:51:36.471]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.471]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c42b14aec3ddbe4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c42b14aec3ddbe4"}
12:51:36 [12:51:36.472]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:8c42b14aec3ddbe4: Error: db error: Error: db error
12:51:36 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:8c42b14aec3ddbe4"}
12:51:36 [12:51:36.473]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.473]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5bce1cdd471d18a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5bce1cdd471d18a1"}
12:51:36 [12:51:36.473]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5bce1cdd471d18a1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5bce1cdd471d18a1"}
12:51:36 [12:51:36.474]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5bce1cdd471d18a1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5bce1cdd471d18a1"}
12:51:36 [12:51:36.475]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.475]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77e6ff544345cafb epochNumber=1
{"provingJobId":"1:BASE_PARITY:77e6ff544345cafb"}
12:51:36 [12:51:36.475]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:77e6ff544345cafb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:77e6ff544345cafb"}
12:51:36 [12:51:36.475]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:77e6ff544345cafb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:77e6ff544345cafb"}
12:51:36 [12:51:36.476]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:77e6ff544345cafb: Error: db error
12:51:36 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:77e6ff544345cafb"}
12:51:36 [12:51:36.477]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.477]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10b6e1343f66783b epochNumber=1
{"provingJobId":"1:BASE_PARITY:10b6e1343f66783b"}
12:51:36 [12:51:36.477]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:10b6e1343f66783b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:10b6e1343f66783b"}
12:51:36 [12:51:36.478]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:10b6e1343f66783b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:10b6e1343f66783b"}
12:51:36 [12:51:36.479]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.479]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4471516cc17a976 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4471516cc17a976"}
12:51:36 [12:51:36.479]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f4471516cc17a976 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f4471516cc17a976"}
12:51:36 [12:51:36.480]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f4471516cc17a976 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f4471516cc17a976"}
12:51:36 [12:51:36.480]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:f4471516cc17a976 jobErr=test error: Error: db error
12:51:36 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:f4471516cc17a976"}
12:51:36 [12:51:36.481]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.482]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dc60b6dd1690c101 not found
{"provingJobId":"1:BASE_PARITY:dc60b6dd1690c101"}
12:51:36 [12:51:36.483]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.483]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:45b703aafeb5271d err=test error
{"provingJoId":"1:BASE_PARITY:45b703aafeb5271d"}
12:51:36 [12:51:36.484]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.494]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b40bc2ec5e77962e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b40bc2ec5e77962e"}
12:51:36 [12:51:36.494]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b40bc2ec5e77962e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b40bc2ec5e77962e"}
12:51:36 [12:51:36.495]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b40bc2ec5e77962e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b40bc2ec5e77962e"}
12:51:36 [12:51:36.505]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:08f6215c51cb3354 epochNumber=2
{"provingJobId":"2:BASE_PARITY:08f6215c51cb3354"}
12:51:36 [12:51:36.506]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:08f6215c51cb3354 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:08f6215c51cb3354"}
12:51:36 [12:51:36.506]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:08f6215c51cb3354 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:08f6215c51cb3354"}
12:51:36 [12:51:36.568]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:9b22e2e12078d1a0 epochNumber=3
{"provingJobId":"3:BASE_PARITY:9b22e2e12078d1a0"}
12:51:36 [12:51:36.585]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:36 [12:51:36.629]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:c78b734de17e0149 epochNumber=4
{"provingJobId":"4:BASE_PARITY:c78b734de17e0149"}
12:51:36 [12:51:36.635]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:36 [12:51:36.690]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:ff054db7974a8094 epochNumber=5
{"provingJobId":"5:BASE_PARITY:ff054db7974a8094"}
12:51:36 [12:51:36.711]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:36 [12:51:36.746]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.747]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85e657cfcd9fc852 epochNumber=42
{"provingJobId":"1:BASE_PARITY:85e657cfcd9fc852"}
12:51:36 [12:51:36.749]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testIsiU2h/42 with map size 1073741824
12:51:36 [12:51:36.753]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:36 [12:51:36.782]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:9a2aa89fe712e734 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9a2aa89fe712e734"}
12:51:36 [12:51:36.789]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.789]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aad073b6b39960d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aad073b6b39960d2"}
12:51:36 [12:51:36.790]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testb2ZoaN/1 with map size 1073741824
12:51:36 [12:51:36.794]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:36 [12:51:36.815]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2255a4a5a0e66f5f epochNumber=1
{"provingJobId":"1:BASE_PARITY:2255a4a5a0e66f5f"}
12:51:36 [12:51:36.824]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.825]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20d1b99ba22a12c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:20d1b99ba22a12c0"}
12:51:36 [12:51:36.826]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzPL5NF/1 with map size 1073741824
12:51:36 [12:51:36.827]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:36 [12:51:36.848]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:20d1b99ba22a12c0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:20d1b99ba22a12c0"}
12:51:36 [12:51:36.852]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.853]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44300633628fe540 epochNumber=1
{"provingJobId":"1:BASE_PARITY:44300633628fe540"}
12:51:36 [12:51:36.854]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testh3fwF5/1 with map size 1073741824
12:51:36 [12:51:36.855]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:36 [12:51:36.875]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:44300633628fe540 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:44300633628fe540"}
12:51:36 [12:51:36.876]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:44300633628fe540 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:44300633628fe540"}
12:51:36 [12:51:36.883]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:44300633628fe540 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:44300633628fe540"}
12:51:36 [12:51:36.887]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.888]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90a609a2d8709d85 epochNumber=1
{"provingJobId":"1:BASE_PARITY:90a609a2d8709d85"}
12:51:36 [12:51:36.888]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3KCSf1/1 with map size 1073741824
12:51:36 [12:51:36.890]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:36 [12:51:36.913]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.918]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.919]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26e9b561fdaef116 epochNumber=1
{"provingJobId":"1:BASE_PARITY:26e9b561fdaef116"}
12:51:36 [12:51:36.919]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmpSGP9/1 with map size 1073741824
12:51:36 [12:51:36.922]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:36 [12:51:36.943]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:26e9b561fdaef116
{"provingJobId":"1:BASE_PARITY:26e9b561fdaef116"}
12:51:36 [12:51:36.943]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:26e9b561fdaef116 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:26e9b561fdaef116"}
12:51:36 [12:51:36.943]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:26e9b561fdaef116 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:26e9b561fdaef116"}
12:51:36 [12:51:36.952]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.953]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15b34384bdb1075a epochNumber=1
{"provingJobId":"1:BASE_PARITY:15b34384bdb1075a"}
12:51:36 [12:51:36.954]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzSRHgu/1 with map size 1073741824
12:51:36 [12:51:36.956]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:36 [12:51:36.975]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:15b34384bdb1075a
{"provingJobId":"1:BASE_PARITY:15b34384bdb1075a"}
12:51:36 [12:51:36.975]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:15b34384bdb1075a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:15b34384bdb1075a"}
12:51:36 [12:51:36.986]
INFO:
prover-client:proving-broker Proving Broker started
12:51:36 [12:51:36.986]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7c4718376a4afe1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d7c4718376a4afe1"}
12:51:36 [12:51:36.987]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testts7dbo/1 with map size 1073741824
12:51:36 [12:51:36.995]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.017]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d7c4718376a4afe1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d7c4718376a4afe1"}
12:51:37 [12:51:37.018]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d7c4718376a4afe1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d7c4718376a4afe1"}
12:51:37 [12:51:37.030]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.031]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c76b37ce3c0479d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c76b37ce3c0479d4"}
12:51:37 [12:51:37.031]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4bnWrC/1 with map size 1073741824
12:51:37 [12:51:37.034]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.056]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c76b37ce3c0479d4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c76b37ce3c0479d4"}
12:51:37 [12:51:37.057]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c76b37ce3c0479d4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c76b37ce3c0479d4"}
12:51:37 [12:51:37.072]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80da4d77f24d25ed epochNumber=0
{"provingJobId":"1:BASE_PARITY:80da4d77f24d25ed"}
12:51:37 [12:51:37.073]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testJL20pU/0 with map size 1073741824
12:51:37 [12:51:37.079]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.100]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:80da4d77f24d25ed epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:80da4d77f24d25ed"}
12:51:37 [12:51:37.101]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22dbefcb5c23c2ba epochNumber=0
{"provingJobId":"1:BASE_PARITY:22dbefcb5c23c2ba"}
12:51:37 [12:51:37.102]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:80da4d77f24d25ed epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:80da4d77f24d25ed"}
12:51:37 [12:51:37.102]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e926c45565623e27 epochNumber=0
{"provingJobId":"1:BASE_PARITY:e926c45565623e27"}
12:51:37 [12:51:37.115]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.118]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.118]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:adcaa6cca85d349d epochNumber=2
{"provingJobId":"1:BASE_PARITY:adcaa6cca85d349d"}
12:51:37 [12:51:37.119]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWTqMTI/2 with map size 1073741824
12:51:37 [12:51:37.121]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.138]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8849805fa3faada1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8849805fa3faada1"}
12:51:37 [12:51:37.139]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWTqMTI/1 with map size 1073741824
12:51:37 [12:51:37.141]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.162]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.163]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86ee434e12f3fc65 epochNumber=1
{"provingJobId":"1:BASE_PARITY:86ee434e12f3fc65"}
12:51:37 [12:51:37.163]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeuxZ8g/1 with map size 1073741824
12:51:37 [12:51:37.165]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.187]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.188]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67fac863fedcff82 epochNumber=1
{"provingJobId":"1:BASE_PARITY:67fac863fedcff82"}
12:51:37 [12:51:37.189]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test02sToB/1 with map size 1073741824
12:51:37 [12:51:37.191]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c32ad6e1b574585 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c32ad6e1b574585"}
12:51:37 [12:51:37.214]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e10c5a4d160e46b epochNumber=2
{"provingJobId":"1:BASE_PARITY:7e10c5a4d160e46b"}
12:51:37 [12:51:37.215]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test02sToB/2 with map size 1073741824
12:51:37 [12:51:37.218]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6b594926676eb75 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b6b594926676eb75"}
12:51:37 [12:51:37.248]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.249]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d54861f13161afe0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d54861f13161afe0"}
12:51:37 [12:51:37.250]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpNjOGZ/1 with map size 1073741824
12:51:37 [12:51:37.251]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.270]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a6c181dcf5872ce epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a6c181dcf5872ce"}
12:51:37 [12:51:37.276]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67ca56e75cf86265 epochNumber=2
{"provingJobId":"1:BASE_PARITY:67ca56e75cf86265"}
12:51:37 [12:51:37.277]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpNjOGZ/2 with map size 1073741824
12:51:37 [12:51:37.280]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.296]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a90caa17de896441 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a90caa17de896441"}
12:51:37 [12:51:37.308]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.309]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3b40d0668f3f338 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e3b40d0668f3f338"}
12:51:37 [12:51:37.310]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVDbRH4/1 with map size 1073741824
12:51:37 [12:51:37.312]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:229e71f907fb2772 epochNumber=1
{"provingJobId":"1:BASE_PARITY:229e71f907fb2772"}
12:51:37 [12:51:37.337]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82fd9e67aab0a3d5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:82fd9e67aab0a3d5"}
12:51:37 [12:51:37.338]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVDbRH4/2 with map size 1073741824
12:51:37 [12:51:37.340]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f621738adcd47c37 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f621738adcd47c37"}
12:51:37 [12:51:37.367]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.368]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b46b7ae1def05c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b46b7ae1def05c6"}
12:51:37 [12:51:37.368]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3061dS/1 with map size 1073741824
12:51:37 [12:51:37.370]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.387]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2f48009ba8baf1a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d2f48009ba8baf1a"}
12:51:37 [12:51:37.394]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2b46b7ae1def05c6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2b46b7ae1def05c6"}
12:51:37 [12:51:37.410]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.412]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05354552a8855cb0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:05354552a8855cb0"}
12:51:37 [12:51:37.412]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUz87mw/1 with map size 1073741824
12:51:37 [12:51:37.414]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.433]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd0dfe2e2b3c163b epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd0dfe2e2b3c163b"}
12:51:37 [12:51:37.440]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:05354552a8855cb0 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:05354552a8855cb0"}
12:51:37 [12:51:37.452]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b67bad029c8f21c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b67bad029c8f21c0"}
12:51:37 [12:51:37.454]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSxeiTW/1 with map size 1073741824
12:51:37 [12:51:37.456]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.474]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae4510eb4e08fe55 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae4510eb4e08fe55"}
12:51:37 [12:51:37.481]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b67bad029c8f21c0 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:b67bad029c8f21c0"}
12:51:37 [12:51:37.486]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.487]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca52494b1a69cd66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca52494b1a69cd66"}
12:51:37 [12:51:37.488]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testN12Y3c/1 with map size 1073741824
12:51:37 [12:51:37.490]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.507]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ca52494b1a69cd66
{"provingJobId":"1:BASE_PARITY:ca52494b1a69cd66"}
12:51:37 [12:51:37.508]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ca52494b1a69cd66 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ca52494b1a69cd66"}
12:51:37 [12:51:37.516]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d986559dba1e45b epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d986559dba1e45b"}
12:51:37 [12:51:37.523]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ca52494b1a69cd66 has already been completed
{"provingJobId":"1:BASE_PARITY:ca52494b1a69cd66"}
12:51:37 [12:51:37.529]
INFO:
prover-client:proving-broker Proving Broker started
12:51:37 [12:51:37.530]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:170d530fc4654851 epochNumber=1
{"provingJobId":"1:BASE_PARITY:170d530fc4654851"}
12:51:37 [12:51:37.531]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeg4jAp/1 with map size 1073741824
12:51:37 [12:51:37.535]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:37 [12:51:37.551]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:390fd638b8f15c3b epochNumber=2
{"provingJobId":"1:BASE_PARITY:390fd638b8f15c3b"}
12:51:37 [12:51:37.552]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testeg4jAp/2 with map size 1073741824
12:51:37 [12:51:37.555]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:38 [12:51:38.625]
INFO:
prover-client:proving-broker Proving Broker started
12:51:38 [12:51:38.633]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:170d530fc4654851 settled=false
{"provingJobId":"1:BASE_PARITY:170d530fc4654851","status":"pending"}
12:51:38 [12:51:38.634]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:390fd638b8f15c3b settled=false
{"provingJobId":"1:BASE_PARITY:390fd638b8f15c3b","status":"pending"}
12:51:38 [12:51:38.635]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:170d530fc4654851 type=BASE_PARITY startedAt=1750423897574 older agent has taken job
{"provingJobId":"1:BASE_PARITY:170d530fc4654851"}
12:51:38 [12:51:38.635]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:170d530fc4654851 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:170d530fc4654851"}
12:51:38 [12:51:38.642]
INFO:
prover-client:proving-broker Proving Broker started
12:51:38 [12:51:38.643]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35f32bbe9d0c7579 epochNumber=1
{"provingJobId":"1:BASE_PARITY:35f32bbe9d0c7579"}
12:51:38 [12:51:38.647]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnphuxR/1 with map size 1073741824
12:51:38 [12:51:38.649]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:38 [12:51:38.671]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cab774d77083ebd7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cab774d77083ebd7"}
12:51:38 [12:51:38.672]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnphuxR/2 with map size 1073741824
12:51:38 [12:51:38.674]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:39 [12:51:39.693]
INFO:
prover-client:proving-broker Proving Broker started
12:51:39 [12:51:39.696]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:35f32bbe9d0c7579 settled=false
{"provingJobId":"1:BASE_PARITY:35f32bbe9d0c7579","status":"pending"}
12:51:39 [12:51:39.699]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cab774d77083ebd7 settled=false
{"provingJobId":"1:BASE_PARITY:cab774d77083ebd7","status":"pending"}
12:51:39 [12:51:39.701]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:35f32bbe9d0c7579 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:35f32bbe9d0c7579"}
12:51:39 [12:51:39.708]
INFO:
prover-client:proving-broker Proving Broker started
12:51:39 [12:51:39.709]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d6292da7e60e788 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d6292da7e60e788"}
12:51:39 [12:51:39.710]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPZrZKA/1 with map size 1073741824
12:51:39 [12:51:39.712]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:39 [12:51:39.730]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:820141462c7c892b epochNumber=2
{"provingJobId":"1:BASE_PARITY:820141462c7c892b"}
12:51:39 [12:51:39.731]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testPZrZKA/2 with map size 1073741824
12:51:39 [12:51:39.734]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:40 [12:51:40.757]
INFO:
prover-client:proving-broker Proving Broker started
12:51:40 [12:51:40.760]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4d6292da7e60e788 settled=false
{"provingJobId":"1:BASE_PARITY:4d6292da7e60e788","status":"pending"}
12:51:40 [12:51:40.762]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:820141462c7c892b settled=false
{"provingJobId":"1:BASE_PARITY:820141462c7c892b","status":"pending"}
12:51:40 [12:51:40.764]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4d6292da7e60e788 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4d6292da7e60e788"}
12:51:40 [12:51:40.764]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4d6292da7e60e788 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4d6292da7e60e788"}
12:51:40 [12:51:40.775]
INFO:
prover-client:proving-broker Proving Broker started
12:51:40 [12:51:40.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d03792a003bce01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d03792a003bce01"}
12:51:40 [12:51:40.776]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqIEnj4/1 with map size 1073741824
12:51:40 [12:51:40.778]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:40 [12:51:40.794]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a8d13b33c4626cf6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a8d13b33c4626cf6"}
12:51:40 [12:51:40.795]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqIEnj4/2 with map size 1073741824
12:51:40 [12:51:40.797]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:40 [12:51:40.814]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8d03792a003bce01 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8d03792a003bce01"}
12:51:40 [12:51:40.820]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a8d13b33c4626cf6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a8d13b33c4626cf6"}
12:51:40 [12:51:40.830]
INFO:
prover-client:proving-broker Proving Broker started
12:51:40 [12:51:40.831]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae73c1fb59e48d16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae73c1fb59e48d16"}
12:51:40 [12:51:40.831]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testP9QF8w/1 with map size 1073741824
12:51:40 [12:51:40.833]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:40 [12:51:40.849]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f258aef67d1f973 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0f258aef67d1f973"}
12:51:40 [12:51:40.851]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testP9QF8w/2 with map size 1073741824
12:51:40 [12:51:40.853]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:40 [12:51:40.867]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ae73c1fb59e48d16 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ae73c1fb59e48d16"}
12:51:40 [12:51:40.868]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ae73c1fb59e48d16 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ae73c1fb59e48d16"}
12:51:40 [12:51:40.873]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0f258aef67d1f973 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0f258aef67d1f973"}
12:51:40 [12:51:40.882]
INFO:
prover-client:proving-broker Proving Broker started
12:51:40 [12:51:40.883]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:484bd3e5aa44d2db err=test error
{"provingJoId":"1:BASE_PARITY:484bd3e5aa44d2db"}
12:51:40 [12:51:40.886]
INFO:
prover-client:proving-broker Proving Broker started
12:51:40 [12:51:40.887]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b84a6d132190e7c3 not found
{"provingJobId":"1:BASE_PARITY:b84a6d132190e7c3"}
12:51:40 [12:51:40.890]
INFO:
prover-client:proving-broker Proving Broker started
12:51:40 [12:51:40.891]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f5a80793ce3d704 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f5a80793ce3d704"}
12:51:40 [12:51:40.891]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testE57mf1/1 with map size 1073741824
12:51:40 [12:51:40.893]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:40 [12:51:40.914]
INFO:
prover-client:proving-broker Proving Broker started
12:51:40 [12:51:40.915]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c63884105a263a16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c63884105a263a16"}
12:51:40 [12:51:40.917]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQqFOGD/1 with map size 1073741824
12:51:40 [12:51:40.919]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.042]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c63884105a263a16 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c63884105a263a16"}
12:51:41 [12:51:41.048]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.049]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4cfc33b9df74aae8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4cfc33b9df74aae8"}
12:51:41 [12:51:41.051]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeqsmV1/1 with map size 1073741824
12:51:41 [12:51:41.053]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.174]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4cfc33b9df74aae8 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4cfc33b9df74aae8"}
12:51:41 [12:51:41.181]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e94666493a50358 epochNumber=10
{"provingJobId":"1:BASE_PARITY:1e94666493a50358"}
12:51:41 [12:51:41.181]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testeqsmV1/10 with map size 1073741824
12:51:41 [12:51:41.184]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.200]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:41 [12:51:41.201]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:51:41 [12:51:41.329]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb9562d000ec881a epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb9562d000ec881a"}
12:51:41 [12:51:41.332]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxgIPgr/1 with map size 1073741824
12:51:41 [12:51:41.334]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.507]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb9562d000ec881a timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:eb9562d000ec881a"}
12:51:41 [12:51:41.560]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.561]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ab0d0d8b8f6c1f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ab0d0d8b8f6c1f9"}
12:51:41 [12:51:41.562]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuPv3Fu/1 with map size 1073741824
12:51:41 [12:51:41.564]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.586]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6ab0d0d8b8f6c1f9 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6ab0d0d8b8f6c1f9"}
12:51:41 [12:51:41.593]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9005f27ce64d5f58 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9005f27ce64d5f58"}
12:51:41 [12:51:41.595]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxwel49/1 with map size 1073741824
12:51:41 [12:51:41.599]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.619]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9005f27ce64d5f58 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:9005f27ce64d5f58"}
12:51:41 [12:51:41.620]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9005f27ce64d5f58 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:9005f27ce64d5f58"}
12:51:41 [12:51:41.631]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.632]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1297a65eebc641e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1297a65eebc641e0"}
12:51:41 [12:51:41.633]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHPucXM/1 with map size 1073741824
12:51:41 [12:51:41.635]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.666]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1297a65eebc641e0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1297a65eebc641e0"}
12:51:41 [12:51:41.678]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.680]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1b19f40e653a1a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1b19f40e653a1a7"}
12:51:41 [12:51:41.681]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwzJ9TA/1 with map size 1073741824
12:51:41 [12:51:41.683]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.708]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d1b19f40e653a1a7 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d1b19f40e653a1a7"}
12:51:41 [12:51:41.709]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26684339bc118b5c epochNumber=3
{"provingJobId":"1:BASE_PARITY:26684339bc118b5c"}
12:51:41 [12:51:41.709]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testwzJ9TA/3 with map size 1073741824
12:51:41 [12:51:41.714]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.755]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:41 [12:51:41.756]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:51:41 [12:51:41.758]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d1b19f40e653a1a7 err=test error
{"provingJoId":"1:BASE_PARITY:d1b19f40e653a1a7"}
12:51:41 [12:51:41.764]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testarTpVE/1 with map size 1073741824
12:51:41 [12:51:41.766]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.787]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testarTpVE/2 with map size 1073741824
12:51:41 [12:51:41.788]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.817]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.820]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:57959df005dece69 settled=false
{"provingJobId":"1:BASE_PARITY:57959df005dece69","status":"pending"}
12:51:41 [12:51:41.822]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7e0235c1bcb6aff7 settled=false
{"provingJobId":"1:BASE_PARITY:7e0235c1bcb6aff7","status":"pending"}
12:51:41 [12:51:41.829]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnnAUAv/1 with map size 1073741824
12:51:41 [12:51:41.832]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.860]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnnAUAv/2 with map size 1073741824
12:51:41 [12:51:41.864]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.896]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.899]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ec26013e596ae2f7 settled=true
{"provingJobId":"1:BASE_PARITY:ec26013e596ae2f7","status":"fulfilled"}
12:51:41 [12:51:41.902]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:24d51ca8069c0463 settled=true
{"provingJobId":"2:BASE_PARITY:24d51ca8069c0463","status":"fulfilled"}
12:51:41 [12:51:41.909]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGM89nw/1 with map size 1073741824
12:51:41 [12:51:41.911]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.935]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGM89nw/2 with map size 1073741824
12:51:41 [12:51:41.943]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.962]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.964]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:68363887ee7d3ace settled=true
{"provingJobId":"1:BASE_PARITY:68363887ee7d3ace","status":"fulfilled"}
12:51:41 [12:51:41.965]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:414944201aecf126 settled=false
{"provingJobId":"1:BASE_PARITY:414944201aecf126","status":"pending"}
12:51:41 [12:51:41.970]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5ca2c4ca2eddad1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5ca2c4ca2eddad1"}
12:51:41 [12:51:41.972]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testS6pQYM/1 with map size 1073741824
12:51:41 [12:51:41.973]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:41 [12:51:41.999]
INFO:
prover-client:proving-broker Proving Broker started
12:51:41 [12:51:41.999]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:970a3de0b9ef7e66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:970a3de0b9ef7e66"}
12:51:42 [12:51:42.000]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:970a3de0b9ef7e66: Error: db error: Error: db error
12:51:42 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:970a3de0b9ef7e66"}
12:51:42 [12:51:42.005]
INFO:
prover-client:proving-broker Proving Broker started
12:51:42 [12:51:42.005]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e354e221dc40642 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e354e221dc40642"}
12:51:42 [12:51:42.006]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test13xiGP/1 with map size 1073741824
12:51:42 [12:51:42.009]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:42 [12:51:42.031]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8e354e221dc40642 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8e354e221dc40642"}
12:51:42 [12:51:42.031]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8e354e221dc40642 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8e354e221dc40642"}
12:51:42 [12:51:42.050]
INFO:
prover-client:proving-broker Proving Broker started
12:51:42 [12:51:42.051]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb2c27deabab4ddf epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb2c27deabab4ddf"}
12:51:42 [12:51:42.055]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testV8BnnR/1 with map size 1073741824
12:51:42 [12:51:42.060]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:42 [12:51:42.083]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb2c27deabab4ddf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:eb2c27deabab4ddf"}
12:51:42 [12:51:42.084]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:eb2c27deabab4ddf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:eb2c27deabab4ddf"}
12:51:42 [12:51:42.085]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:eb2c27deabab4ddf: Error: db error
12:51:42 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:eb2c27deabab4ddf"}
12:51:42 [12:51:42.088]
INFO:
prover-client:proving-broker Proving Broker started
12:51:42 [12:51:42.089]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b6653ee0a2cec74 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b6653ee0a2cec74"}
12:51:42 [12:51:42.090]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testETVR8S/1 with map size 1073741824
12:51:42 [12:51:42.094]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:42 [12:51:42.111]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3b6653ee0a2cec74 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3b6653ee0a2cec74"}
12:51:42 [12:51:42.111]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3b6653ee0a2cec74 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3b6653ee0a2cec74"}
12:51:42 [12:51:42.124]
INFO:
prover-client:proving-broker Proving Broker started
12:51:42 [12:51:42.125]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ab36d531e91ae66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1ab36d531e91ae66"}
12:51:42 [12:51:42.127]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIOx8LU/1 with map size 1073741824
12:51:42 [12:51:42.129]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:42 [12:51:42.154]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1ab36d531e91ae66 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1ab36d531e91ae66"}
12:51:42 [12:51:42.155]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1ab36d531e91ae66 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1ab36d531e91ae66"}
12:51:42 [12:51:42.155]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:1ab36d531e91ae66 jobErr=test error: Error: db error
12:51:42 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:1ab36d531e91ae66"}
12:51:42 [12:51:42.160]
INFO:
prover-client:proving-broker Proving Broker started
12:51:42 [12:51:42.160]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:290dc4083fc76057 not found
{"provingJobId":"1:BASE_PARITY:290dc4083fc76057"}
12:51:42 [12:51:42.163]
INFO:
prover-client:proving-broker Proving Broker started
12:51:42 [12:51:42.164]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:681e9de541fed851 err=test error
{"provingJoId":"1:BASE_PARITY:681e9de541fed851"}
12:51:42 [12:51:42.166]
INFO:
prover-client:proving-broker Proving Broker started
12:51:42 [12:51:42.178]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:558970798dca77b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:558970798dca77b6"}
12:51:42 [12:51:42.181]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdJ2X5y/1 with map size 1073741824
12:51:42 [12:51:42.186]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:42 [12:51:42.205]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:558970798dca77b6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:558970798dca77b6"}
12:51:42 [12:51:42.206]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:558970798dca77b6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:558970798dca77b6"}
12:51:42 [12:51:42.223]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:1efc9c2c6ab2660d epochNumber=2
{"provingJobId":"2:BASE_PARITY:1efc9c2c6ab2660d"}
12:51:42 [12:51:42.224]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testdJ2X5y/2 with map size 1073741824
12:51:42 [12:51:42.227]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:42 [12:51:42.263]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:1efc9c2c6ab2660d type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:1efc9c2c6ab2660d"}
12:51:42 [12:51:42.264]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:1efc9c2c6ab2660d type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:1efc9c2c6ab2660d"}
12:51:42 [12:51:42.333]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:f23e7d67e6174c49 epochNumber=3
{"provingJobId":"3:BASE_PARITY:f23e7d67e6174c49"}
12:51:42 [12:51:42.336]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testdJ2X5y/3 with map size 1073741824
12:51:42 [12:51:42.350]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:42 [12:51:42.372]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:42 [12:51:42.373]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:51:42 [12:51:42.433]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:142253666d24c5ef epochNumber=4
{"provingJobId":"4:BASE_PARITY:142253666d24c5ef"}
12:51:42 [12:51:42.434]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testdJ2X5y/4 with map size 1073741824
12:51:42 [12:51:42.444]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:42 [12:51:42.476]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:42 [12:51:42.477]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
12:51:42 [12:51:42.526]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:86d537a4305eaca6 epochNumber=5
{"provingJobId":"5:BASE_PARITY:86d537a4305eaca6"}
12:51:42 [12:51:42.527]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testdJ2X5y/5 with map size 1073741824
12:51:42 [12:51:42.530]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:51:42 [12:51:42.557]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:51:42 [12:51:42.557]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
12:51:42
PASS src/proving_broker/proving_broker.test.ts (
13.235 s)
12:51:42 ProvingBroker
12:51:42 Producer API
12:51:42
✓ refuses stale jobs (43 ms)
12:51:42
✓ enqueues jobs (4 ms)
12:51:42
✓ ignores duplicate jobs (3 ms)
12:51:42
✓ reports correct status when enqueuing repeat jobs (3 ms)
12:51:42
✓ throws an error in case of duplicate job IDs (25 ms)
12:51:42
✓ returns not-found status for non-existing jobs (1 ms)
12:51:42
✓ cancels jobs in queue (3 ms)
12:51:42
✓ cancels jobs in-progress (2 ms)
12:51:42
✓ returns job result if successful (2 ms)
12:51:42
✓ returns job error if failed (2 ms)
12:51:42
✓ correctly returns job status for concurrent writes (3 ms)
12:51:42
✓ refuses stale jobs (42 ms)
12:51:42
✓ enqueues jobs (36 ms)
12:51:42
✓ ignores duplicate jobs (27 ms)
12:51:42
✓ reports correct status when enqueuing repeat jobs (34 ms)
12:51:42
✓ throws an error in case of duplicate job IDs (26 ms)
12:51:42
✓ returns not-found status for non-existing jobs (3 ms)
12:51:42
✓ cancels jobs in queue (36 ms)
12:51:42
✓ cancels jobs in-progress (33 ms)
12:51:42
✓ returns job result if successful (43 ms)
12:51:42
✓ returns job error if failed (41 ms)
12:51:42
✓ correctly returns job status for concurrent writes (43 ms)
12:51:42 Consumer API
12:51:42
✓ returns undefined if no jobs are available (10 ms)
12:51:42
✓ returns jobs in priority order (2 ms)
12:51:42
✓ returns undefined if no jobs are available for the given allowList (2 ms)
12:51:42
✓ returns a job if it is in the allowList (4 ms)
12:51:42
✓ returns the most important job if it is in the allowList (3 ms)
12:51:42
✓ returns any job if filter is empty (2 ms)
12:51:42
✓ returns a new job when reporting job success (3 ms)
12:51:42
✓ returns a new job when reporting permanent error (2 ms)
12:51:42
✓ returns a new job when reporting retry-able error (2 ms)
12:51:42
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
12:51:42
✓ returns a new job if job is already in progress elsewhere (1059 ms)
12:51:42
✓ avoids sending the same job to a new agent after a restart (1007 ms)
12:51:42
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
12:51:42
✓ tracks job result if in progress (2 ms)
12:51:42
✓ tracks job result even if job is in queue (2 ms)
12:51:42
✓ ignores reported job error if unknown job (1 ms)
12:51:42
✓ ignores job result if unknown job (1 ms)
12:51:42
✓ returns undefined if no jobs are available (2 ms)
12:51:42
✓ returns jobs in priority order (44 ms)
12:51:42
✓ returns undefined if no jobs are available for the given allowList (25 ms)
12:51:42
✓ returns a job if it is in the allowList (60 ms)
12:51:42
✓ returns the most important job if it is in the allowList (59 ms)
12:51:42
✓ returns any job if filter is empty (59 ms)
12:51:42
✓ returns a new job when reporting job success (38 ms)
12:51:42
✓ returns a new job when reporting permanent error (46 ms)
12:51:42
✓ returns a new job when reporting retry-able error (33 ms)
12:51:42
✓ returns a new job when reporting progress if current one is cancelled (42 ms)
12:51:42
✓ returns a new job if job is already in progress elsewhere (1114 ms)
12:51:42
✓ avoids sending the same job to a new agent after a restart (1065 ms)
12:51:42
✓ avoids sending a completed job to a new agent after a restart (1066 ms)
12:51:42
✓ tracks job result if in progress (55 ms)
12:51:42
✓ tracks job result even if job is in queue (52 ms)
12:51:42
✓ ignores reported job error if unknown job (4 ms)
12:51:42
✓ ignores job result if unknown job (3 ms)
12:51:42 Timeouts
12:51:42
✓ tracks in progress jobs (1 ms)
12:51:42
✓ re-enqueues jobs that time out (109 ms)
12:51:42
✓ cancel stale jobs that time out (236 ms)
12:51:42
✓ keeps the jobs in progress while it is alive (205 ms)
12:51:42
✓ tracks in progress jobs (23 ms)
12:51:42
✓ re-enqueues jobs that time out (134 ms)
12:51:42
✓ cancel stale jobs that time out (281 ms)
12:51:42
✓ keeps the jobs in progress while it is alive (229 ms)
12:51:42 Retries
12:51:42
✓ retries jobs (4 ms)
12:51:42
✓ retries up to a maximum number of times (3 ms)
12:51:42
✓ passing retry=false does not retry (2 ms)
12:51:42
✓ does not retry if job is stale (28 ms)
12:51:42
✓ retries jobs (32 ms)
12:51:42
✓ retries up to a maximum number of times (40 ms)
12:51:42
✓ passing retry=false does not retry (47 ms)
12:51:42
✓ does not retry if job is stale (84 ms)
12:51:42 Database management
12:51:42
✓ re-enqueues proof requests on start (3 ms)
12:51:42
✓ restores proof results on start (2 ms)
12:51:42
✓ only re-enqueues unfinished jobs (2 ms)
12:51:42
✓ saves job when enqueued (2 ms)
12:51:42
✓ does not retain job if database fails to save (2 ms)
12:51:42
✓ saves job result (2 ms)
12:51:42
✓ saves result even if database fails to save (2 ms)
12:51:42
✓ saves job error (2 ms)
12:51:42
✓ saves job error even if database fails to save (3 ms)
12:51:42
✓ does not save job result if job is unknown (1 ms)
12:51:42
✓ does not save job error if job is unknown (1 ms)
12:51:42
✓ cleans up old jobs periodically (259 ms)
12:51:42
✓ re-enqueues proof requests on start (64 ms)
12:51:42
✓ restores proof results on start (79 ms)
12:51:42
✓ only re-enqueues unfinished jobs (61 ms)
12:51:42
✓ saves job when enqueued (28 ms)
12:51:42
✓ does not retain job if database fails to save (4 ms)
12:51:42
✓ saves job result (40 ms)
12:51:42
✓ saves result even if database fails to save (43 ms)
12:51:42
✓ saves job error (33 ms)
12:51:42
✓ saves job error even if database fails to save (36 ms)
12:51:42
✓ does not save job result if job is unknown (3 ms)
12:51:42
✓ does not save job error if job is unknown (4 ms)
12:51:42
✓ cleans up old jobs periodically (442 ms)
12:51:42
12:51:42
Test Suites: 1 passed, 1 total
12:51:42
Tests: 96 passed, 96 total
12:51:42
Snapshots: 0 total
12:51:42
Time: 13.332 s
12:51:42
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
12:51:42
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?