Parent Log:
http://ci.aztec-labs.com/6379f596748997ca
Command: 2ca1986c0c6d5c2c yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/9222c7777effe26aa04b5367a455d25be0e7f62e
Env: REF_NAME=gh-readonly-queue/next/pr-15055-a9bb2f683797c8e631b892c5f539a2867bddaabf CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Wed Jun 18 16:02:46 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15055_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
16:02:51 [16:02:51.830]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.836]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:962cdd5ba7a4fbb0 epochNumber=42
{"provingJobId":"1:BASE_PARITY:962cdd5ba7a4fbb0"}
16:02:51 [16:02:51.840]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:1f37cd2cb2f654b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f37cd2cb2f654b3"}
16:02:51 [16:02:51.875]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.876]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c15fdf0ae44963d epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c15fdf0ae44963d"}
16:02:51 [16:02:51.877]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7ed9a4d9fda1830 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7ed9a4d9fda1830"}
16:02:51 [16:02:51.878]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.879]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc9d1c93e78c6400 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc9d1c93e78c6400"}
16:02:51 [16:02:51.880]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dc9d1c93e78c6400 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dc9d1c93e78c6400"}
16:02:51 [16:02:51.882]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.884]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a3d6a100de7e45f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a3d6a100de7e45f"}
16:02:51 [16:02:51.885]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4a3d6a100de7e45f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4a3d6a100de7e45f"}
16:02:51 [16:02:51.886]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4a3d6a100de7e45f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4a3d6a100de7e45f"}
16:02:51 [16:02:51.887]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4a3d6a100de7e45f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4a3d6a100de7e45f"}
16:02:51 [16:02:51.888]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.888]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:629ff6c8bd0143c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:629ff6c8bd0143c3"}
16:02:51 [16:02:51.910]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.912]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.913]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6a3fd4047120474 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6a3fd4047120474"}
16:02:51 [16:02:51.914]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c6a3fd4047120474
{"provingJobId":"1:BASE_PARITY:c6a3fd4047120474"}
16:02:51 [16:02:51.914]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c6a3fd4047120474 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c6a3fd4047120474"}
16:02:51 [16:02:51.915]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c6a3fd4047120474 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c6a3fd4047120474"}
16:02:51 [16:02:51.917]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.917]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8eeb94f8fe25dda3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8eeb94f8fe25dda3"}
16:02:51 [16:02:51.918]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:8eeb94f8fe25dda3
{"provingJobId":"1:BASE_PARITY:8eeb94f8fe25dda3"}
16:02:51 [16:02:51.919]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8eeb94f8fe25dda3 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:8eeb94f8fe25dda3"}
16:02:51 [16:02:51.921]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.921]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c3fe677f2de8214 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7c3fe677f2de8214"}
16:02:51 [16:02:51.922]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7c3fe677f2de8214 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7c3fe677f2de8214"}
16:02:51 [16:02:51.923]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7c3fe677f2de8214 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7c3fe677f2de8214"}
16:02:51 [16:02:51.925]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.925]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9b5cd77b1559107 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9b5cd77b1559107"}
16:02:51 [16:02:51.926]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b9b5cd77b1559107 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b9b5cd77b1559107"}
16:02:51 [16:02:51.926]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b9b5cd77b1559107 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b9b5cd77b1559107"}
16:02:51 [16:02:51.928]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.928]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc40443b9ff44bc8 epochNumber=0
{"provingJobId":"1:BASE_PARITY:fc40443b9ff44bc8"}
16:02:51 [16:02:51.928]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fc40443b9ff44bc8 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fc40443b9ff44bc8"}
16:02:51 [16:02:51.929]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61584abf6f8966e5 epochNumber=0
{"provingJobId":"1:BASE_PARITY:61584abf6f8966e5"}
16:02:51 [16:02:51.929]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fc40443b9ff44bc8 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fc40443b9ff44bc8"}
16:02:51 [16:02:51.929]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba5e4cc4f4ba56a3 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ba5e4cc4f4ba56a3"}
16:02:51 [16:02:51.931]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.932]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.932]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:65874344d14f679e epochNumber=2
{"provingJobId":"1:BASE_PARITY:65874344d14f679e"}
16:02:51 [16:02:51.933]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f29bc2fe16a7ea26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f29bc2fe16a7ea26"}
16:02:51 [16:02:51.934]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.934]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b84371b1e269087d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b84371b1e269087d"}
16:02:51 [16:02:51.935]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.936]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1addda079dad5841 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1addda079dad5841"}
16:02:51 [16:02:51.936]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61dee46bd6f0bc4d epochNumber=1
{"provingJobId":"1:BASE_PARITY:61dee46bd6f0bc4d"}
16:02:51 [16:02:51.936]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94d3af91e125c1e4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:94d3af91e125c1e4"}
16:02:51 [16:02:51.936]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cdfb52f37186d928 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cdfb52f37186d928"}
16:02:51 [16:02:51.937]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.937]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1f0f7a4fbd4afbe epochNumber=1
{"provingJobId":"1:BASE_PARITY:c1f0f7a4fbd4afbe"}
16:02:51 [16:02:51.938]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:27f880bb035ea500 epochNumber=1
{"provingJobId":"1:BASE_PARITY:27f880bb035ea500"}
16:02:51 [16:02:51.938]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c6ca7ef8d2ae54d epochNumber=2
{"provingJobId":"1:BASE_PARITY:7c6ca7ef8d2ae54d"}
16:02:51 [16:02:51.938]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19ea9683e862cb38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:19ea9683e862cb38"}
16:02:51 [16:02:51.939]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa152baf6ec8a457 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa152baf6ec8a457"}
16:02:51 [16:02:51.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e93cb82c26dd6b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e93cb82c26dd6b0"}
16:02:51 [16:02:51.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b59917d58668a4c6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b59917d58668a4c6"}
16:02:51 [16:02:51.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80e31d394650f626 epochNumber=1
{"provingJobId":"1:BASE_PARITY:80e31d394650f626"}
16:02:51 [16:02:51.941]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.942]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc867ea4fa3b8e6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc867ea4fa3b8e6a"}
16:02:51 [16:02:51.942]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c68d0ff220bcec2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c68d0ff220bcec2"}
16:02:51 [16:02:51.942]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dc867ea4fa3b8e6a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dc867ea4fa3b8e6a"}
16:02:51 [16:02:51.943]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d5cb8f0683e50196 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d5cb8f0683e50196"}
16:02:51 [16:02:51.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:705a5e836a8be141 epochNumber=1
{"provingJobId":"1:BASE_PARITY:705a5e836a8be141"}
16:02:51 [16:02:51.945]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d5cb8f0683e50196 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:d5cb8f0683e50196"}
16:02:51 [16:02:51.945]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.947]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b39a7cf695014ddc epochNumber=1
{"provingJobId":"1:BASE_PARITY:b39a7cf695014ddc"}
16:02:51 [16:02:51.947]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:836a18994e304d91 epochNumber=1
{"provingJobId":"1:BASE_PARITY:836a18994e304d91"}
16:02:51 [16:02:51.948]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b39a7cf695014ddc type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:b39a7cf695014ddc"}
16:02:51 [16:02:51.957]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.958]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ef1cfb9d307ad5f epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ef1cfb9d307ad5f"}
16:02:51 [16:02:51.959]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7ef1cfb9d307ad5f
{"provingJobId":"1:BASE_PARITY:7ef1cfb9d307ad5f"}
16:02:51 [16:02:51.959]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7ef1cfb9d307ad5f type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7ef1cfb9d307ad5f"}
16:02:51 [16:02:51.960]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:717c295d477e8b8b epochNumber=1
{"provingJobId":"1:BASE_PARITY:717c295d477e8b8b"}
16:02:51 [16:02:51.961]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7ef1cfb9d307ad5f has already been completed
{"provingJobId":"1:BASE_PARITY:7ef1cfb9d307ad5f"}
16:02:51 [16:02:51.962]
INFO:
prover-client:proving-broker Proving Broker started
16:02:51 [16:02:51.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b4372b856cc7cff epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b4372b856cc7cff"}
16:02:51 [16:02:51.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51289f1e1f8e7d62 epochNumber=2
{"provingJobId":"1:BASE_PARITY:51289f1e1f8e7d62"}
16:02:53 [16:02:53.015]
INFO:
prover-client:proving-broker Proving Broker started
16:02:53 [16:02:53.016]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7b4372b856cc7cff settled=false
{"provingJobId":"1:BASE_PARITY:7b4372b856cc7cff","status":"pending"}
16:02:53 [16:02:53.016]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:51289f1e1f8e7d62 settled=false
{"provingJobId":"1:BASE_PARITY:51289f1e1f8e7d62","status":"pending"}
16:02:53 [16:02:53.017]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b4372b856cc7cff type=BASE_PARITY startedAt=1750262571963 older agent has taken job
{"provingJobId":"1:BASE_PARITY:7b4372b856cc7cff"}
16:02:53 [16:02:53.018]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b4372b856cc7cff type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:7b4372b856cc7cff"}
16:02:53 [16:02:53.019]
INFO:
prover-client:proving-broker Proving Broker started
16:02:53 [16:02:53.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd617134be5690f1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd617134be5690f1"}
16:02:53 [16:02:53.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a66b73d99d5bac5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3a66b73d99d5bac5"}
16:02:54 [16:02:54.021]
INFO:
prover-client:proving-broker Proving Broker started
16:02:54 [16:02:54.022]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dd617134be5690f1 settled=false
{"provingJobId":"1:BASE_PARITY:dd617134be5690f1","status":"pending"}
16:02:54 [16:02:54.022]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3a66b73d99d5bac5 settled=false
{"provingJobId":"1:BASE_PARITY:3a66b73d99d5bac5","status":"pending"}
16:02:54 [16:02:54.023]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dd617134be5690f1 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:dd617134be5690f1"}
16:02:54 [16:02:54.025]
INFO:
prover-client:proving-broker Proving Broker started
16:02:54 [16:02:54.027]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc64918106f77795 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc64918106f77795"}
16:02:54 [16:02:54.027]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e09594936726e58 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8e09594936726e58"}
16:02:55 [16:02:55.028]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.030]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dc64918106f77795 settled=false
{"provingJobId":"1:BASE_PARITY:dc64918106f77795","status":"pending"}
16:02:55 [16:02:55.031]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8e09594936726e58 settled=false
{"provingJobId":"1:BASE_PARITY:8e09594936726e58","status":"pending"}
16:02:55 [16:02:55.032]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dc64918106f77795 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dc64918106f77795"}
16:02:55 [16:02:55.032]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dc64918106f77795 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dc64918106f77795"}
16:02:55 [16:02:55.035]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.035]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b4e0345469ed2cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b4e0345469ed2cb"}
16:02:55 [16:02:55.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd9fe1b399b5e1d1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:bd9fe1b399b5e1d1"}
16:02:55 [16:02:55.037]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6b4e0345469ed2cb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6b4e0345469ed2cb"}
16:02:55 [16:02:55.037]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bd9fe1b399b5e1d1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bd9fe1b399b5e1d1"}
16:02:55 [16:02:55.038]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.039]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9c7d852f0569736 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9c7d852f0569736"}
16:02:55 [16:02:55.039]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e368e0ea09d84080 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e368e0ea09d84080"}
16:02:55 [16:02:55.039]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e9c7d852f0569736 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e9c7d852f0569736"}
16:02:55 [16:02:55.040]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e9c7d852f0569736 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e9c7d852f0569736"}
16:02:55 [16:02:55.040]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e368e0ea09d84080 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e368e0ea09d84080"}
16:02:55 [16:02:55.041]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.042]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2784b20537a3e59c err=test error
{"provingJoId":"1:BASE_PARITY:2784b20537a3e59c"}
16:02:55 [16:02:55.043]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.043]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a1e497809875b0a3 not found
{"provingJobId":"1:BASE_PARITY:a1e497809875b0a3"}
16:02:55 [16:02:55.045]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.045]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bfe119d4378a8254 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bfe119d4378a8254"}
16:02:55 [16:02:55.046]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.048]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:774b6eb49ab481fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:774b6eb49ab481fa"}
16:02:55 [16:02:55.173]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:774b6eb49ab481fa timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:774b6eb49ab481fa"}
16:02:55 [16:02:55.176]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.177]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b8116e1650bc0dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b8116e1650bc0dc"}
16:02:55 [16:02:55.278]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3b8116e1650bc0dc timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:3b8116e1650bc0dc"}
16:02:55 [16:02:55.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce41cc4b640c3ff2 epochNumber=10
{"provingJobId":"1:BASE_PARITY:ce41cc4b640c3ff2"}
16:02:55 [16:02:55.304]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:02:55 [16:02:55.416]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:691052d0241b56d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:691052d0241b56d2"}
16:02:55 [16:02:55.594]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:691052d0241b56d2 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:691052d0241b56d2"}
16:02:55 [16:02:55.625]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.626]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c237dede9ce0f06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c237dede9ce0f06"}
16:02:55 [16:02:55.627]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1c237dede9ce0f06 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1c237dede9ce0f06"}
16:02:55 [16:02:55.629]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.630]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d143d0544548af2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d143d0544548af2"}
16:02:55 [16:02:55.630]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6d143d0544548af2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6d143d0544548af2"}
16:02:55 [16:02:55.631]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6d143d0544548af2 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:6d143d0544548af2"}
16:02:55 [16:02:55.633]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.634]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8d224455813b573 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8d224455813b573"}
16:02:55 [16:02:55.634]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b8d224455813b573 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b8d224455813b573"}
16:02:55 [16:02:55.636]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.636]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d53473a316867dce epochNumber=1
{"provingJobId":"1:BASE_PARITY:d53473a316867dce"}
16:02:55 [16:02:55.637]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d53473a316867dce type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d53473a316867dce"}
16:02:55 [16:02:55.637]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f5cccd808009cae epochNumber=3
{"provingJobId":"1:BASE_PARITY:4f5cccd808009cae"}
16:02:55 [16:02:55.661]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:02:55 [16:02:55.663]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d53473a316867dce err=test error
{"provingJoId":"1:BASE_PARITY:d53473a316867dce"}
16:02:55 [16:02:55.665]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.666]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1fe1ac6f5d179845 settled=false
{"provingJobId":"1:BASE_PARITY:1fe1ac6f5d179845","status":"pending"}
16:02:55 [16:02:55.666]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6b7678c452cc6bb6 settled=false
{"provingJobId":"1:BASE_PARITY:6b7678c452cc6bb6","status":"pending"}
16:02:55 [16:02:55.668]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.669]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a3b82f79e70e7c6b settled=true
{"provingJobId":"1:BASE_PARITY:a3b82f79e70e7c6b","status":"fulfilled"}
16:02:55 [16:02:55.669]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:957e7c516ea5af27 settled=true
{"provingJobId":"2:BASE_PARITY:957e7c516ea5af27","status":"fulfilled"}
16:02:55 [16:02:55.670]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.671]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2ac3c89fc9ef1e2c settled=true
{"provingJobId":"1:BASE_PARITY:2ac3c89fc9ef1e2c","status":"fulfilled"}
16:02:55 [16:02:55.671]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5d09e63b869b31d7 settled=false
{"provingJobId":"1:BASE_PARITY:5d09e63b869b31d7","status":"pending"}
16:02:55 [16:02:55.672]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.674]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8aca4132c2dbd570 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8aca4132c2dbd570"}
16:02:55 [16:02:55.676]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.676]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1aed29db6bf96a8e epochNumber=1
{"provingJobId":"1:BASE_PARITY:1aed29db6bf96a8e"}
16:02:55 [16:02:55.677]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:1aed29db6bf96a8e: Error: db error: Error: db error
16:02:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:1aed29db6bf96a8e"}
16:02:55 [16:02:55.679]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.679]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf039daf62654742 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf039daf62654742"}
16:02:55 [16:02:55.679]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bf039daf62654742 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bf039daf62654742"}
16:02:55 [16:02:55.680]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bf039daf62654742 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bf039daf62654742"}
16:02:55 [16:02:55.681]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.681]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fd17df79b30a75a epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fd17df79b30a75a"}
16:02:55 [16:02:55.682]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0fd17df79b30a75a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0fd17df79b30a75a"}
16:02:55 [16:02:55.682]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0fd17df79b30a75a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0fd17df79b30a75a"}
16:02:55 [16:02:55.683]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:0fd17df79b30a75a: Error: db error
16:02:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:0fd17df79b30a75a"}
16:02:55 [16:02:55.684]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.684]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9dc9ce8f5cdd21ab epochNumber=1
{"provingJobId":"1:BASE_PARITY:9dc9ce8f5cdd21ab"}
16:02:55 [16:02:55.685]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9dc9ce8f5cdd21ab type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9dc9ce8f5cdd21ab"}
16:02:55 [16:02:55.685]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9dc9ce8f5cdd21ab type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9dc9ce8f5cdd21ab"}
16:02:55 [16:02:55.686]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.687]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72ef71af0c241f0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:72ef71af0c241f0b"}
16:02:55 [16:02:55.687]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:72ef71af0c241f0b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:72ef71af0c241f0b"}
16:02:55 [16:02:55.687]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:72ef71af0c241f0b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:72ef71af0c241f0b"}
16:02:55 [16:02:55.688]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:72ef71af0c241f0b jobErr=test error: Error: db error
16:02:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:72ef71af0c241f0b"}
16:02:55 [16:02:55.689]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.690]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:727d5a6a41598479 not found
{"provingJobId":"1:BASE_PARITY:727d5a6a41598479"}
16:02:55 [16:02:55.691]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.692]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:96be29f2acb33181 err=test error
{"provingJoId":"1:BASE_PARITY:96be29f2acb33181"}
16:02:55 [16:02:55.693]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.703]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17072a62ee463bdb epochNumber=1
{"provingJobId":"1:BASE_PARITY:17072a62ee463bdb"}
16:02:55 [16:02:55.704]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:17072a62ee463bdb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:17072a62ee463bdb"}
16:02:55 [16:02:55.704]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:17072a62ee463bdb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:17072a62ee463bdb"}
16:02:55 [16:02:55.714]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:c0f1b9949efd474b epochNumber=2
{"provingJobId":"2:BASE_PARITY:c0f1b9949efd474b"}
16:02:55 [16:02:55.715]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:c0f1b9949efd474b type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:c0f1b9949efd474b"}
16:02:55 [16:02:55.715]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c0f1b9949efd474b type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:c0f1b9949efd474b"}
16:02:55 [16:02:55.777]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:1ff0eb8bf3744cf5 epochNumber=3
{"provingJobId":"3:BASE_PARITY:1ff0eb8bf3744cf5"}
16:02:55 [16:02:55.794]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:02:55 [16:02:55.839]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:7b7e2d4915be7ff7 epochNumber=4
{"provingJobId":"4:BASE_PARITY:7b7e2d4915be7ff7"}
16:02:55 [16:02:55.847]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:02:55 [16:02:55.901]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:eff2fc6d38f1196c epochNumber=5
{"provingJobId":"5:BASE_PARITY:eff2fc6d38f1196c"}
16:02:55 [16:02:55.925]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:02:55 [16:02:55.956]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.957]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b8a14526d45ddae epochNumber=42
{"provingJobId":"1:BASE_PARITY:3b8a14526d45ddae"}
16:02:55 [16:02:55.958]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-test4fY7UZ/42 with map size 1073741824
16:02:55 [16:02:55.963]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:55 [16:02:55.988]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:a7c6b9838b37cfc2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a7c6b9838b37cfc2"}
16:02:55 [16:02:55.993]
INFO:
prover-client:proving-broker Proving Broker started
16:02:55 [16:02:55.993]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b346dec93e055c3b epochNumber=1
{"provingJobId":"1:BASE_PARITY:b346dec93e055c3b"}
16:02:55 [16:02:55.994]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOjtjx4/1 with map size 1073741824
16:02:55 [16:02:55.996]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.013]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5cee8885ebfae88b epochNumber=1
{"provingJobId":"1:BASE_PARITY:5cee8885ebfae88b"}
16:02:56 [16:02:56.023]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.024]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cdf3f8294b606f18 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cdf3f8294b606f18"}
16:02:56 [16:02:56.025]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzpaJlV/1 with map size 1073741824
16:02:56 [16:02:56.027]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.044]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:cdf3f8294b606f18 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:cdf3f8294b606f18"}
16:02:56 [16:02:56.048]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.049]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75bdc5772fd9c8ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:75bdc5772fd9c8ba"}
16:02:56 [16:02:56.050]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testleqGCA/1 with map size 1073741824
16:02:56 [16:02:56.051]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.068]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:75bdc5772fd9c8ba epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:75bdc5772fd9c8ba"}
16:02:56 [16:02:56.069]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:75bdc5772fd9c8ba type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:75bdc5772fd9c8ba"}
16:02:56 [16:02:56.076]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:75bdc5772fd9c8ba epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:75bdc5772fd9c8ba"}
16:02:56 [16:02:56.080]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.081]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ebb4dd17828fb674 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ebb4dd17828fb674"}
16:02:56 [16:02:56.082]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3ti41V/1 with map size 1073741824
16:02:56 [16:02:56.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.118]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.124]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d83f45e31c8884d epochNumber=1
{"provingJobId":"1:BASE_PARITY:2d83f45e31c8884d"}
16:02:56 [16:02:56.125]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLOS9ro/1 with map size 1073741824
16:02:56 [16:02:56.128]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.147]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2d83f45e31c8884d
{"provingJobId":"1:BASE_PARITY:2d83f45e31c8884d"}
16:02:56 [16:02:56.148]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2d83f45e31c8884d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2d83f45e31c8884d"}
16:02:56 [16:02:56.148]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2d83f45e31c8884d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2d83f45e31c8884d"}
16:02:56 [16:02:56.157]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.158]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a81846e91b0ba953 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a81846e91b0ba953"}
16:02:56 [16:02:56.158]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEXc32i/1 with map size 1073741824
16:02:56 [16:02:56.162]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.182]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a81846e91b0ba953
{"provingJobId":"1:BASE_PARITY:a81846e91b0ba953"}
16:02:56 [16:02:56.182]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a81846e91b0ba953 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a81846e91b0ba953"}
16:02:56 [16:02:56.191]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.193]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b22ecf5208aca8f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b22ecf5208aca8f9"}
16:02:56 [16:02:56.194]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfvty4o/1 with map size 1073741824
16:02:56 [16:02:56.197]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.217]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b22ecf5208aca8f9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b22ecf5208aca8f9"}
16:02:56 [16:02:56.218]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b22ecf5208aca8f9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b22ecf5208aca8f9"}
16:02:56 [16:02:56.230]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.231]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fd7630047f8eaa3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fd7630047f8eaa3"}
16:02:56 [16:02:56.232]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrcyG4k/1 with map size 1073741824
16:02:56 [16:02:56.235]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.253]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0fd7630047f8eaa3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0fd7630047f8eaa3"}
16:02:56 [16:02:56.253]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0fd7630047f8eaa3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0fd7630047f8eaa3"}
16:02:56 [16:02:56.262]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.263]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea16f0a140f325d9 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ea16f0a140f325d9"}
16:02:56 [16:02:56.263]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testXUGAyy/0 with map size 1073741824
16:02:56 [16:02:56.266]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.283]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ea16f0a140f325d9 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ea16f0a140f325d9"}
16:02:56 [16:02:56.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f5a9acfd2ed9c3a epochNumber=0
{"provingJobId":"1:BASE_PARITY:7f5a9acfd2ed9c3a"}
16:02:56 [16:02:56.285]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ea16f0a140f325d9 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ea16f0a140f325d9"}
16:02:56 [16:02:56.285]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c8c384ba645988a epochNumber=0
{"provingJobId":"1:BASE_PARITY:7c8c384ba645988a"}
16:02:56 [16:02:56.300]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.303]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.304]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f42306d846caf0fb epochNumber=2
{"provingJobId":"1:BASE_PARITY:f42306d846caf0fb"}
16:02:56 [16:02:56.306]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testF0wmuC/2 with map size 1073741824
16:02:56 [16:02:56.313]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c48ae476b34dc24b epochNumber=1
{"provingJobId":"1:BASE_PARITY:c48ae476b34dc24b"}
16:02:56 [16:02:56.347]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testF0wmuC/1 with map size 1073741824
16:02:56 [16:02:56.352]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.377]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.378]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3d8d8e1931c7312 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3d8d8e1931c7312"}
16:02:56 [16:02:56.378]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt8UqwU/1 with map size 1073741824
16:02:56 [16:02:56.384]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.416]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4eeda3b3bc4410d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4eeda3b3bc4410d5"}
16:02:56 [16:02:56.418]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKtiMpU/1 with map size 1073741824
16:02:56 [16:02:56.420]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.446]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7336a5ec302574f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7336a5ec302574f5"}
16:02:56 [16:02:56.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98a106e93d2993c8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:98a106e93d2993c8"}
16:02:56 [16:02:56.455]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKtiMpU/2 with map size 1073741824
16:02:56 [16:02:56.457]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.476]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e85a16f68a0deb3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e85a16f68a0deb3"}
16:02:56 [16:02:56.486]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.487]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25336c98b42dee41 epochNumber=1
{"provingJobId":"1:BASE_PARITY:25336c98b42dee41"}
16:02:56 [16:02:56.488]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9Meys6/1 with map size 1073741824
16:02:56 [16:02:56.490]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78e9f1dcbb1c5fcd epochNumber=1
{"provingJobId":"1:BASE_PARITY:78e9f1dcbb1c5fcd"}
16:02:56 [16:02:56.515]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf0212347814e682 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cf0212347814e682"}
16:02:56 [16:02:56.517]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test9Meys6/2 with map size 1073741824
16:02:56 [16:02:56.520]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.536]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e2b945549cd5056 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e2b945549cd5056"}
16:02:56 [16:02:56.546]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.546]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53e2edb968813c26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53e2edb968813c26"}
16:02:56 [16:02:56.547]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfCHKNK/1 with map size 1073741824
16:02:56 [16:02:56.549]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.569]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca5cdcdeda1949c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca5cdcdeda1949c6"}
16:02:56 [16:02:56.575]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e28e5281cfec21d epochNumber=2
{"provingJobId":"1:BASE_PARITY:6e28e5281cfec21d"}
16:02:56 [16:02:56.576]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfCHKNK/2 with map size 1073741824
16:02:56 [16:02:56.578]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:804093515d2b59ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:804093515d2b59ec"}
16:02:56 [16:02:56.605]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64060182990c4f5c epochNumber=1
{"provingJobId":"1:BASE_PARITY:64060182990c4f5c"}
16:02:56 [16:02:56.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJOIiCx/1 with map size 1073741824
16:02:56 [16:02:56.609]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.633]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6dce03b086784fc4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6dce03b086784fc4"}
16:02:56 [16:02:56.642]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:64060182990c4f5c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:64060182990c4f5c"}
16:02:56 [16:02:56.654]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.655]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8bf19f838279a9c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8bf19f838279a9c7"}
16:02:56 [16:02:56.655]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGPWiHP/1 with map size 1073741824
16:02:56 [16:02:56.657]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.675]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e6f626315373262 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e6f626315373262"}
16:02:56 [16:02:56.680]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8bf19f838279a9c7 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:8bf19f838279a9c7"}
16:02:56 [16:02:56.692]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4df851b685d62792 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4df851b685d62792"}
16:02:56 [16:02:56.693]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testj07YoB/1 with map size 1073741824
16:02:56 [16:02:56.700]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.725]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cb1929db13f4787 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0cb1929db13f4787"}
16:02:56 [16:02:56.735]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4df851b685d62792 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:4df851b685d62792"}
16:02:56 [16:02:56.740]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41ae5b79435d2b45 epochNumber=1
{"provingJobId":"1:BASE_PARITY:41ae5b79435d2b45"}
16:02:56 [16:02:56.742]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlCHBa9/1 with map size 1073741824
16:02:56 [16:02:56.744]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.761]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:41ae5b79435d2b45
{"provingJobId":"1:BASE_PARITY:41ae5b79435d2b45"}
16:02:56 [16:02:56.761]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:41ae5b79435d2b45 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:41ae5b79435d2b45"}
16:02:56 [16:02:56.768]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c5f96da4e92e94f epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c5f96da4e92e94f"}
16:02:56 [16:02:56.775]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:41ae5b79435d2b45 has already been completed
{"provingJobId":"1:BASE_PARITY:41ae5b79435d2b45"}
16:02:56 [16:02:56.780]
INFO:
prover-client:proving-broker Proving Broker started
16:02:56 [16:02:56.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:389054ca4c0c7566 epochNumber=1
{"provingJobId":"1:BASE_PARITY:389054ca4c0c7566"}
16:02:56 [16:02:56.783]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7mMqRX/1 with map size 1073741824
16:02:56 [16:02:56.785]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:56 [16:02:56.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17f9a21e2f73e738 epochNumber=2
{"provingJobId":"1:BASE_PARITY:17f9a21e2f73e738"}
16:02:56 [16:02:56.803]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7mMqRX/2 with map size 1073741824
16:02:56 [16:02:56.806]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:57 [16:02:57.877]
INFO:
prover-client:proving-broker Proving Broker started
16:02:57 [16:02:57.884]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:389054ca4c0c7566 settled=false
{"provingJobId":"1:BASE_PARITY:389054ca4c0c7566","status":"pending"}
16:02:57 [16:02:57.886]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:17f9a21e2f73e738 settled=false
{"provingJobId":"1:BASE_PARITY:17f9a21e2f73e738","status":"pending"}
16:02:57 [16:02:57.887]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:389054ca4c0c7566 type=BASE_PARITY startedAt=1750262576825 older agent has taken job
{"provingJobId":"1:BASE_PARITY:389054ca4c0c7566"}
16:02:57 [16:02:57.887]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:389054ca4c0c7566 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:389054ca4c0c7566"}
16:02:57 [16:02:57.891]
INFO:
prover-client:proving-broker Proving Broker started
16:02:57 [16:02:57.891]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7915c009bc4e73ee epochNumber=1
{"provingJobId":"1:BASE_PARITY:7915c009bc4e73ee"}
16:02:57 [16:02:57.895]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzZJIIj/1 with map size 1073741824
16:02:57 [16:02:57.896]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:57 [16:02:57.912]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21356171a0a673f8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:21356171a0a673f8"}
16:02:57 [16:02:57.913]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testzZJIIj/2 with map size 1073741824
16:02:57 [16:02:57.915]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:58 [16:02:58.931]
INFO:
prover-client:proving-broker Proving Broker started
16:02:58 [16:02:58.933]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7915c009bc4e73ee settled=false
{"provingJobId":"1:BASE_PARITY:7915c009bc4e73ee","status":"pending"}
16:02:58 [16:02:58.935]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:21356171a0a673f8 settled=false
{"provingJobId":"1:BASE_PARITY:21356171a0a673f8","status":"pending"}
16:02:58 [16:02:58.936]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7915c009bc4e73ee type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:7915c009bc4e73ee"}
16:02:58 [16:02:58.940]
INFO:
prover-client:proving-broker Proving Broker started
16:02:58 [16:02:58.941]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0276d855490b6fbe epochNumber=1
{"provingJobId":"1:BASE_PARITY:0276d855490b6fbe"}
16:02:58 [16:02:58.941]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGJ8JfC/1 with map size 1073741824
16:02:58 [16:02:58.943]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:58 [16:02:58.959]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03993488dac8b27e epochNumber=2
{"provingJobId":"1:BASE_PARITY:03993488dac8b27e"}
16:02:58 [16:02:58.960]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGJ8JfC/2 with map size 1073741824
16:02:58 [16:02:58.962]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:02:59 [16:02:59.979]
INFO:
prover-client:proving-broker Proving Broker started
16:02:59 [16:02:59.982]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0276d855490b6fbe settled=false
{"provingJobId":"1:BASE_PARITY:0276d855490b6fbe","status":"pending"}
16:02:59 [16:02:59.985]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:03993488dac8b27e settled=false
{"provingJobId":"1:BASE_PARITY:03993488dac8b27e","status":"pending"}
16:02:59 [16:02:59.987]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0276d855490b6fbe type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0276d855490b6fbe"}
16:02:59 [16:02:59.987]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0276d855490b6fbe type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0276d855490b6fbe"}
16:02:59 [16:02:59.999]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:764de38292ef2624 epochNumber=1
{"provingJobId":"1:BASE_PARITY:764de38292ef2624"}
16:03:00 [16:03:00.001]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testahT84m/1 with map size 1073741824
16:03:00 [16:03:00.004]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a2d2b7be1276308 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8a2d2b7be1276308"}
16:03:00 [16:03:00.032]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testahT84m/2 with map size 1073741824
16:03:00 [16:03:00.036]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.061]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:764de38292ef2624 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:764de38292ef2624"}
16:03:00 [16:03:00.067]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8a2d2b7be1276308 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8a2d2b7be1276308"}
16:03:00 [16:03:00.076]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1cdb501d646565a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1cdb501d646565a6"}
16:03:00 [16:03:00.079]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYIHZmG/1 with map size 1073741824
16:03:00 [16:03:00.084]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.104]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b4c11494150f1f8a epochNumber=2
{"provingJobId":"1:BASE_PARITY:b4c11494150f1f8a"}
16:03:00 [16:03:00.105]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYIHZmG/2 with map size 1073741824
16:03:00 [16:03:00.107]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.128]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1cdb501d646565a6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1cdb501d646565a6"}
16:03:00 [16:03:00.128]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1cdb501d646565a6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1cdb501d646565a6"}
16:03:00 [16:03:00.136]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b4c11494150f1f8a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b4c11494150f1f8a"}
16:03:00 [16:03:00.149]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.150]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:70f79f300901c195 err=test error
{"provingJoId":"1:BASE_PARITY:70f79f300901c195"}
16:03:00 [16:03:00.156]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.157]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5564f0d97314e34f not found
{"provingJobId":"1:BASE_PARITY:5564f0d97314e34f"}
16:03:00 [16:03:00.160]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.161]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe9f022807327421 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe9f022807327421"}
16:03:00 [16:03:00.162]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test00gdhh/1 with map size 1073741824
16:03:00 [16:03:00.165]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.189]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.190]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1eb71845bac45f90 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1eb71845bac45f90"}
16:03:00 [16:03:00.191]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test17wDoh/1 with map size 1073741824
16:03:00 [16:03:00.193]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.318]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1eb71845bac45f90 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1eb71845bac45f90"}
16:03:00 [16:03:00.326]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.327]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77db05e4f79ee091 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77db05e4f79ee091"}
16:03:00 [16:03:00.329]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testK87k9I/1 with map size 1073741824
16:03:00 [16:03:00.330]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.453]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:77db05e4f79ee091 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:77db05e4f79ee091"}
16:03:00 [16:03:00.457]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8909989277f5cfee epochNumber=10
{"provingJobId":"1:BASE_PARITY:8909989277f5cfee"}
16:03:00 [16:03:00.459]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testK87k9I/10 with map size 1073741824
16:03:00 [16:03:00.462]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.504]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:03:00 [16:03:00.505]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:03:00 [16:03:00.612]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.613]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:981379e462c03cc3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:981379e462c03cc3"}
16:03:00 [16:03:00.614]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoPYVVl/1 with map size 1073741824
16:03:00 [16:03:00.616]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.792]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:981379e462c03cc3 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:981379e462c03cc3"}
16:03:00 [16:03:00.843]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.844]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8390f320823f71d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8390f320823f71d"}
16:03:00 [16:03:00.844]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtphOTF/1 with map size 1073741824
16:03:00 [16:03:00.846]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.870]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b8390f320823f71d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b8390f320823f71d"}
16:03:00 [16:03:00.875]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.876]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b99789026ceed3a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b99789026ceed3a6"}
16:03:00 [16:03:00.876]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKhea2M/1 with map size 1073741824
16:03:00 [16:03:00.878]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.904]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b99789026ceed3a6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b99789026ceed3a6"}
16:03:00 [16:03:00.905]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b99789026ceed3a6 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:b99789026ceed3a6"}
16:03:00 [16:03:00.920]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.921]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d63b654f741d892c epochNumber=1
{"provingJobId":"1:BASE_PARITY:d63b654f741d892c"}
16:03:00 [16:03:00.926]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtxHHcj/1 with map size 1073741824
16:03:00 [16:03:00.929]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.955]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d63b654f741d892c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d63b654f741d892c"}
16:03:00 [16:03:00.967]
INFO:
prover-client:proving-broker Proving Broker started
16:03:00 [16:03:00.968]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:252ff187cb1d8630 epochNumber=1
{"provingJobId":"1:BASE_PARITY:252ff187cb1d8630"}
16:03:00 [16:03:00.969]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRUuLht/1 with map size 1073741824
16:03:00 [16:03:00.972]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:00 [16:03:00.993]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:252ff187cb1d8630 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:252ff187cb1d8630"}
16:03:00 [16:03:00.994]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:145654c45e8e895a epochNumber=3
{"provingJobId":"1:BASE_PARITY:145654c45e8e895a"}
16:03:00 [16:03:00.995]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testRUuLht/3 with map size 1073741824
16:03:00 [16:03:00.997]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.021]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:03:01 [16:03:01.021]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:03:01 [16:03:01.042]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:252ff187cb1d8630 err=test error
{"provingJoId":"1:BASE_PARITY:252ff187cb1d8630"}
16:03:01 [16:03:01.048]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testodT11J/1 with map size 1073741824
16:03:01 [16:03:01.050]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.072]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testodT11J/2 with map size 1073741824
16:03:01 [16:03:01.076]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.105]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.107]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0c28a0fe077ed4b2 settled=false
{"provingJobId":"1:BASE_PARITY:0c28a0fe077ed4b2","status":"pending"}
16:03:01 [16:03:01.112]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7b447058f57bafdc settled=false
{"provingJobId":"1:BASE_PARITY:7b447058f57bafdc","status":"pending"}
16:03:01 [16:03:01.125]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIup6Jx/1 with map size 1073741824
16:03:01 [16:03:01.135]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.166]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIup6Jx/2 with map size 1073741824
16:03:01 [16:03:01.170]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.214]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.219]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5d0620dc150eff41 settled=true
{"provingJobId":"1:BASE_PARITY:5d0620dc150eff41","status":"fulfilled"}
16:03:01 [16:03:01.222]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:aeff2f54cec71998 settled=true
{"provingJobId":"2:BASE_PARITY:aeff2f54cec71998","status":"fulfilled"}
16:03:01 [16:03:01.231]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testViHQlz/1 with map size 1073741824
16:03:01 [16:03:01.234]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.263]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testViHQlz/2 with map size 1073741824
16:03:01 [16:03:01.270]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.297]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.300]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:318ed2716dcd8ec7 settled=true
{"provingJobId":"1:BASE_PARITY:318ed2716dcd8ec7","status":"fulfilled"}
16:03:01 [16:03:01.302]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ace4346cdf603a58 settled=false
{"provingJobId":"1:BASE_PARITY:ace4346cdf603a58","status":"pending"}
16:03:01 [16:03:01.307]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.307]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f17d545fb72f954 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7f17d545fb72f954"}
16:03:01 [16:03:01.308]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMMoWpd/1 with map size 1073741824
16:03:01 [16:03:01.319]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.351]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.352]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ddac09d13028341f epochNumber=1
{"provingJobId":"1:BASE_PARITY:ddac09d13028341f"}
16:03:01 [16:03:01.352]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:ddac09d13028341f: Error: db error: Error: db error
16:03:01 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:ddac09d13028341f"}
16:03:01 [16:03:01.355]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f3cdafb684aedc4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f3cdafb684aedc4"}
16:03:01 [16:03:01.357]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4xraWm/1 with map size 1073741824
16:03:01 [16:03:01.362]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.386]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4f3cdafb684aedc4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4f3cdafb684aedc4"}
16:03:01 [16:03:01.387]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4f3cdafb684aedc4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4f3cdafb684aedc4"}
16:03:01 [16:03:01.400]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.400]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8f1d22eaf03a9ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:f8f1d22eaf03a9ff"}
16:03:01 [16:03:01.402]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIHyosn/1 with map size 1073741824
16:03:01 [16:03:01.404]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.429]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f8f1d22eaf03a9ff type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f8f1d22eaf03a9ff"}
16:03:01 [16:03:01.429]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f8f1d22eaf03a9ff type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f8f1d22eaf03a9ff"}
16:03:01 [16:03:01.430]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:f8f1d22eaf03a9ff: Error: db error
16:03:01 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:f8f1d22eaf03a9ff"}
16:03:01 [16:03:01.436]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e3c026b8f2c7423 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e3c026b8f2c7423"}
16:03:01 [16:03:01.437]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtveBnK/1 with map size 1073741824
16:03:01 [16:03:01.441]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.467]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6e3c026b8f2c7423 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6e3c026b8f2c7423"}
16:03:01 [16:03:01.468]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6e3c026b8f2c7423 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6e3c026b8f2c7423"}
16:03:01 [16:03:01.480]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.480]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1336bc0320da42c epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1336bc0320da42c"}
16:03:01 [16:03:01.481]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQyPzuD/1 with map size 1073741824
16:03:01 [16:03:01.487]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.507]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e1336bc0320da42c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e1336bc0320da42c"}
16:03:01 [16:03:01.508]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e1336bc0320da42c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e1336bc0320da42c"}
16:03:01 [16:03:01.509]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:e1336bc0320da42c jobErr=test error: Error: db error
16:03:01 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:e1336bc0320da42c"}
16:03:01 [16:03:01.517]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.518]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:94832d5cba462700 not found
{"provingJobId":"1:BASE_PARITY:94832d5cba462700"}
16:03:01 [16:03:01.523]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.524]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:911cc6a3e54eeb17 err=test error
{"provingJoId":"1:BASE_PARITY:911cc6a3e54eeb17"}
16:03:01 [16:03:01.529]
INFO:
prover-client:proving-broker Proving Broker started
16:03:01 [16:03:01.540]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e89ba75d1d46d72d epochNumber=1
{"provingJobId":"1:BASE_PARITY:e89ba75d1d46d72d"}
16:03:01 [16:03:01.542]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWWSr4O/1 with map size 1073741824
16:03:01 [16:03:01.544]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.570]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e89ba75d1d46d72d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e89ba75d1d46d72d"}
16:03:01 [16:03:01.571]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e89ba75d1d46d72d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e89ba75d1d46d72d"}
16:03:01 [16:03:01.587]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:ea644c3617d52ed9 epochNumber=2
{"provingJobId":"2:BASE_PARITY:ea644c3617d52ed9"}
16:03:01 [16:03:01.589]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWWSr4O/2 with map size 1073741824
16:03:01 [16:03:01.593]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.614]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:ea644c3617d52ed9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:ea644c3617d52ed9"}
16:03:01 [16:03:01.615]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:ea644c3617d52ed9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:ea644c3617d52ed9"}
16:03:01 [16:03:01.682]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:a386ec9ed93cad3b epochNumber=3
{"provingJobId":"3:BASE_PARITY:a386ec9ed93cad3b"}
16:03:01 [16:03:01.684]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testWWSr4O/3 with map size 1073741824
16:03:01 [16:03:01.686]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.710]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:03:01 [16:03:01.710]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:03:01 [16:03:01.765]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:1b13f107a3d37a10 epochNumber=4
{"provingJobId":"4:BASE_PARITY:1b13f107a3d37a10"}
16:03:01 [16:03:01.766]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testWWSr4O/4 with map size 1073741824
16:03:01 [16:03:01.771]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.790]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:03:01 [16:03:01.790]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
16:03:01 [16:03:01.849]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:906458118d5dcb80 epochNumber=5
{"provingJobId":"5:BASE_PARITY:906458118d5dcb80"}
16:03:01 [16:03:01.850]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testWWSr4O/5 with map size 1073741824
16:03:01 [16:03:01.853]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:03:01 [16:03:01.896]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:03:01 [16:03:01.897]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
16:03:01
PASS src/proving_broker/proving_broker.test.ts (
13.883 s)
16:03:01 ProvingBroker
16:03:01 Producer API
16:03:01
✓ refuses stale jobs (42 ms)
16:03:01
✓ enqueues jobs (4 ms)
16:03:01
✓ ignores duplicate jobs (3 ms)
16:03:01
✓ reports correct status when enqueuing repeat jobs (5 ms)
16:03:01
✓ throws an error in case of duplicate job IDs (21 ms)
16:03:01
✓ returns not-found status for non-existing jobs (1 ms)
16:03:01
✓ cancels jobs in queue (4 ms)
16:03:01
✓ cancels jobs in-progress (4 ms)
16:03:01
✓ returns job result if successful (4 ms)
16:03:01
✓ returns job error if failed (3 ms)
16:03:01
✓ correctly returns job status for concurrent writes (3 ms)
16:03:01
✓ refuses stale jobs (37 ms)
16:03:01
✓ enqueues jobs (30 ms)
16:03:01
✓ ignores duplicate jobs (24 ms)
16:03:01
✓ reports correct status when enqueuing repeat jobs (31 ms)
16:03:01
✓ throws an error in case of duplicate job IDs (36 ms)
16:03:01
✓ returns not-found status for non-existing jobs (7 ms)
16:03:01
✓ cancels jobs in queue (33 ms)
16:03:01
✓ cancels jobs in-progress (33 ms)
16:03:01
✓ returns job result if successful (39 ms)
16:03:01
✓ returns job error if failed (32 ms)
16:03:01
✓ correctly returns job status for concurrent writes (37 ms)
16:03:01 Consumer API
16:03:01
✓ returns undefined if no jobs are available (2 ms)
16:03:01
✓ returns jobs in priority order (2 ms)
16:03:01
✓ returns undefined if no jobs are available for the given allowList (1 ms)
16:03:01
✓ returns a job if it is in the allowList (2 ms)
16:03:01
✓ returns the most important job if it is in the allowList (2 ms)
16:03:01
✓ returns any job if filter is empty (2 ms)
16:03:01
✓ returns a new job when reporting job success (2 ms)
16:03:01
✓ returns a new job when reporting permanent error (2 ms)
16:03:01
✓ returns a new job when reporting retry-able error (3 ms)
16:03:01
✓ returns a new job when reporting progress if current one is cancelled (12 ms)
16:03:01
✓ returns a new job if job is already in progress elsewhere (1057 ms)
16:03:01
✓ avoids sending the same job to a new agent after a restart (1005 ms)
16:03:01
✓ avoids sending a completed job to a new agent after a restart (1008 ms)
16:03:01
✓ tracks job result if in progress (4 ms)
16:03:01
✓ tracks job result even if job is in queue (3 ms)
16:03:01
✓ ignores reported job error if unknown job (1 ms)
16:03:01
✓ ignores job result if unknown job (2 ms)
16:03:01
✓ returns undefined if no jobs are available (2 ms)
16:03:01
✓ returns jobs in priority order (73 ms)
16:03:01
✓ returns undefined if no jobs are available for the given allowList (34 ms)
16:03:01
✓ returns a job if it is in the allowList (73 ms)
16:03:01
✓ returns the most important job if it is in the allowList (59 ms)
16:03:01
✓ returns any job if filter is empty (58 ms)
16:03:01
✓ returns a new job when reporting job success (49 ms)
16:03:01
✓ returns a new job when reporting permanent error (37 ms)
16:03:01
✓ returns a new job when reporting retry-able error (47 ms)
16:03:01
✓ returns a new job when reporting progress if current one is cancelled (40 ms)
16:03:01
✓ returns a new job if job is already in progress elsewhere (1111 ms)
16:03:01
✓ avoids sending the same job to a new agent after a restart (1049 ms)
16:03:01
✓ avoids sending a completed job to a new agent after a restart (1057 ms)
16:03:01
✓ tracks job result if in progress (79 ms)
16:03:01
✓ tracks job result even if job is in queue (71 ms)
16:03:01
✓ ignores reported job error if unknown job (4 ms)
16:03:01
✓ ignores job result if unknown job (6 ms)
16:03:01 Timeouts
16:03:01
✓ tracks in progress jobs (2 ms)
16:03:01
✓ re-enqueues jobs that time out (129 ms)
16:03:01
✓ cancel stale jobs that time out (239 ms)
16:03:01
✓ keeps the jobs in progress while it is alive (209 ms)
16:03:01
✓ tracks in progress jobs (28 ms)
16:03:01
✓ re-enqueues jobs that time out (136 ms)
16:03:01
✓ cancel stale jobs that time out (285 ms)
16:03:01
✓ keeps the jobs in progress while it is alive (229 ms)
16:03:01 Retries
16:03:01
✓ retries jobs (3 ms)
16:03:01
✓ retries up to a maximum number of times (3 ms)
16:03:01
✓ passing retry=false does not retry (3 ms)
16:03:01
✓ does not retry if job is stale (29 ms)
16:03:01
✓ retries jobs (32 ms)
16:03:01
✓ retries up to a maximum number of times (41 ms)
16:03:01
✓ passing retry=false does not retry (50 ms)
16:03:01
✓ does not retry if job is stale (80 ms)
16:03:01 Database management
16:03:01
✓ re-enqueues proof requests on start (3 ms)
16:03:01
✓ restores proof results on start (1 ms)
16:03:01
✓ only re-enqueues unfinished jobs (2 ms)
16:03:01
✓ saves job when enqueued (3 ms)
16:03:01
✓ does not retain job if database fails to save (3 ms)
16:03:01
✓ saves job result (2 ms)
16:03:01
✓ saves result even if database fails to save (2 ms)
16:03:01
✓ saves job error (3 ms)
16:03:01
✓ saves job error even if database fails to save (3 ms)
16:03:01
✓ does not save job result if job is unknown (1 ms)
16:03:01
✓ does not save job error if job is unknown (1 ms)
16:03:01
✓ cleans up old jobs periodically (261 ms)
16:03:01
✓ re-enqueues proof requests on start (71 ms)
16:03:01
✓ restores proof results on start (110 ms)
16:03:01
✓ only re-enqueues unfinished jobs (76 ms)
16:03:01
✓ saves job when enqueued (43 ms)
16:03:01
✓ does not retain job if database fails to save (4 ms)
16:03:01
✓ saves job result (43 ms)
16:03:01
✓ saves result even if database fails to save (35 ms)
16:03:01
✓ saves job error (44 ms)
16:03:01
✓ saves job error even if database fails to save (33 ms)
16:03:01
✓ does not save job result if job is unknown (9 ms)
16:03:01
✓ does not save job error if job is unknown (4 ms)
16:03:01
✓ cleans up old jobs periodically (403 ms)
16:03:01
16:03:01
Test Suites: 1 passed, 1 total
16:03:01
Tests: 96 passed, 96 total
16:03:01
Snapshots: 0 total
16:03:01
Time: 14.005 s
16:03:01
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
16:03:01
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?