Parent Log:
http://ci.aztec-labs.com/176cdc16aad7fcf7
Command: 98e833669a704616 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/d20f79089216edeeff0ae1b191f645fa82ba33b6
Env: REF_NAME=gh-readonly-queue/next/pr-14967-082468f0c1e7e5424b3654b5be0f3560eed7e487 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Thu Jun 12 12:10:04 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14967_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
12:10:08 [12:10:08.174]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.180]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7fdce8f7d0320e09 epochNumber=42
{"provingJobId":"1:BASE_PARITY:7fdce8f7d0320e09"}
12:10:08 [12:10:08.183]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:df2cb8ace8552ee1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df2cb8ace8552ee1"}
12:10:08 [12:10:08.211]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.212]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cdca464f9095eec1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cdca464f9095eec1"}
12:10:08 [12:10:08.213]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ac3c6505281986c epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ac3c6505281986c"}
12:10:08 [12:10:08.214]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.215]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58e20b6909621d2b epochNumber=1
{"provingJobId":"1:BASE_PARITY:58e20b6909621d2b"}
12:10:08 [12:10:08.216]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:58e20b6909621d2b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:58e20b6909621d2b"}
12:10:08 [12:10:08.218]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.218]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77889ee3b1bb69c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77889ee3b1bb69c1"}
12:10:08 [12:10:08.219]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:77889ee3b1bb69c1 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:77889ee3b1bb69c1"}
12:10:08 [12:10:08.220]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:77889ee3b1bb69c1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:77889ee3b1bb69c1"}
12:10:08 [12:10:08.220]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:77889ee3b1bb69c1 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:77889ee3b1bb69c1"}
12:10:08 [12:10:08.221]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f081f67f3950a0a epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f081f67f3950a0a"}
12:10:08 [12:10:08.250]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.251]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.252]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6e5bc1961560e28 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6e5bc1961560e28"}
12:10:08 [12:10:08.253]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a6e5bc1961560e28
{"provingJobId":"1:BASE_PARITY:a6e5bc1961560e28"}
12:10:08 [12:10:08.253]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a6e5bc1961560e28 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a6e5bc1961560e28"}
12:10:08 [12:10:08.254]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a6e5bc1961560e28 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a6e5bc1961560e28"}
12:10:08 [12:10:08.255]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42c6986a73eec6d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:42c6986a73eec6d0"}
12:10:08 [12:10:08.256]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:42c6986a73eec6d0
{"provingJobId":"1:BASE_PARITY:42c6986a73eec6d0"}
12:10:08 [12:10:08.256]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:42c6986a73eec6d0 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:42c6986a73eec6d0"}
12:10:08 [12:10:08.257]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.258]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d152f588fe268fb8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d152f588fe268fb8"}
12:10:08 [12:10:08.258]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d152f588fe268fb8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d152f588fe268fb8"}
12:10:08 [12:10:08.258]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d152f588fe268fb8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d152f588fe268fb8"}
12:10:08 [12:10:08.259]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.259]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1a3124d4e26b722 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1a3124d4e26b722"}
12:10:08 [12:10:08.260]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e1a3124d4e26b722 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e1a3124d4e26b722"}
12:10:08 [12:10:08.260]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e1a3124d4e26b722 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e1a3124d4e26b722"}
12:10:08 [12:10:08.261]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd92bc0786cbcc81 epochNumber=0
{"provingJobId":"1:BASE_PARITY:bd92bc0786cbcc81"}
12:10:08 [12:10:08.261]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:bd92bc0786cbcc81 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:bd92bc0786cbcc81"}
12:10:08 [12:10:08.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5959d850411508f1 epochNumber=0
{"provingJobId":"1:BASE_PARITY:5959d850411508f1"}
12:10:08 [12:10:08.262]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:bd92bc0786cbcc81 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:bd92bc0786cbcc81"}
12:10:08 [12:10:08.262]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36a0c5242a100c14 epochNumber=0
{"provingJobId":"1:BASE_PARITY:36a0c5242a100c14"}
12:10:08 [12:10:08.263]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.263]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.264]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77cc0d09a820ea6b epochNumber=2
{"provingJobId":"1:BASE_PARITY:77cc0d09a820ea6b"}
12:10:08 [12:10:08.264]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a013619656c63c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a013619656c63c8"}
12:10:08 [12:10:08.265]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d28cbf7ceb2cf1ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:d28cbf7ceb2cf1ba"}
12:10:08 [12:10:08.266]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e24c95fc49f1801b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e24c95fc49f1801b"}
12:10:08 [12:10:08.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f17b2d962508021 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f17b2d962508021"}
12:10:08 [12:10:08.267]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c196f7655bb0c6ab epochNumber=2
{"provingJobId":"1:BASE_PARITY:c196f7655bb0c6ab"}
12:10:08 [12:10:08.267]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:169e75a602212949 epochNumber=1
{"provingJobId":"1:BASE_PARITY:169e75a602212949"}
12:10:08 [12:10:08.268]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:241b9bf05c40e875 epochNumber=1
{"provingJobId":"1:BASE_PARITY:241b9bf05c40e875"}
12:10:08 [12:10:08.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:262e0936a911a806 epochNumber=1
{"provingJobId":"1:BASE_PARITY:262e0936a911a806"}
12:10:08 [12:10:08.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a31324949d30451 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0a31324949d30451"}
12:10:08 [12:10:08.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25d6dd535351630b epochNumber=1
{"provingJobId":"1:BASE_PARITY:25d6dd535351630b"}
12:10:08 [12:10:08.278]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3683facf0f46dc7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3683facf0f46dc7"}
12:10:08 [12:10:08.280]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56a0c02e22b90237 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56a0c02e22b90237"}
12:10:08 [12:10:08.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:495b08c62a10fa13 epochNumber=2
{"provingJobId":"1:BASE_PARITY:495b08c62a10fa13"}
12:10:08 [12:10:08.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d022a0d68488a0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d022a0d68488a0b"}
12:10:08 [12:10:08.283]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.283]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9c57f02cb7d305c epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9c57f02cb7d305c"}
12:10:08 [12:10:08.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca58f4d5a4386a3e epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca58f4d5a4386a3e"}
12:10:08 [12:10:08.284]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e9c57f02cb7d305c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e9c57f02cb7d305c"}
12:10:08 [12:10:08.285]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00c0464d3bf0813c epochNumber=1
{"provingJobId":"1:BASE_PARITY:00c0464d3bf0813c"}
12:10:08 [12:10:08.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2ae04713facaf9b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e2ae04713facaf9b"}
12:10:08 [12:10:08.286]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:00c0464d3bf0813c type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:00c0464d3bf0813c"}
12:10:08 [12:10:08.287]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.288]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:353ee23ed6934e40 epochNumber=1
{"provingJobId":"1:BASE_PARITY:353ee23ed6934e40"}
12:10:08 [12:10:08.289]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:168fd05a4ae2d862 epochNumber=1
{"provingJobId":"1:BASE_PARITY:168fd05a4ae2d862"}
12:10:08 [12:10:08.289]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:353ee23ed6934e40 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:353ee23ed6934e40"}
12:10:08 [12:10:08.290]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.291]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a4b27f0cd88469a epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a4b27f0cd88469a"}
12:10:08 [12:10:08.292]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5a4b27f0cd88469a
{"provingJobId":"1:BASE_PARITY:5a4b27f0cd88469a"}
12:10:08 [12:10:08.292]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5a4b27f0cd88469a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5a4b27f0cd88469a"}
12:10:08 [12:10:08.293]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0d8fcf21b49495b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0d8fcf21b49495b"}
12:10:08 [12:10:08.293]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5a4b27f0cd88469a has already been completed
{"provingJobId":"1:BASE_PARITY:5a4b27f0cd88469a"}
12:10:08 [12:10:08.294]
INFO:
prover-client:proving-broker Proving Broker started
12:10:08 [12:10:08.295]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:107f96cbe2dd6977 epochNumber=1
{"provingJobId":"1:BASE_PARITY:107f96cbe2dd6977"}
12:10:08 [12:10:08.295]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f3d7878f7ac2f70 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0f3d7878f7ac2f70"}
12:10:09 [12:10:09.347]
INFO:
prover-client:proving-broker Proving Broker started
12:10:09 [12:10:09.347]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:107f96cbe2dd6977 settled=false
{"provingJobId":"1:BASE_PARITY:107f96cbe2dd6977","status":"pending"}
12:10:09 [12:10:09.348]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0f3d7878f7ac2f70 settled=false
{"provingJobId":"1:BASE_PARITY:0f3d7878f7ac2f70","status":"pending"}
12:10:09 [12:10:09.349]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:107f96cbe2dd6977 type=BASE_PARITY startedAt=1749730208295 older agent has taken job
{"provingJobId":"1:BASE_PARITY:107f96cbe2dd6977"}
12:10:09 [12:10:09.349]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:107f96cbe2dd6977 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:107f96cbe2dd6977"}
12:10:09 [12:10:09.350]
INFO:
prover-client:proving-broker Proving Broker started
12:10:09 [12:10:09.351]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab14ca7a4ece2919 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab14ca7a4ece2919"}
12:10:09 [12:10:09.351]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b56edfff0cb549b epochNumber=2
{"provingJobId":"1:BASE_PARITY:0b56edfff0cb549b"}
12:10:10 [12:10:10.352]
INFO:
prover-client:proving-broker Proving Broker started
12:10:10 [12:10:10.353]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ab14ca7a4ece2919 settled=false
{"provingJobId":"1:BASE_PARITY:ab14ca7a4ece2919","status":"pending"}
12:10:10 [12:10:10.353]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0b56edfff0cb549b settled=false
{"provingJobId":"1:BASE_PARITY:0b56edfff0cb549b","status":"pending"}
12:10:10 [12:10:10.353]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ab14ca7a4ece2919 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:ab14ca7a4ece2919"}
12:10:10 [12:10:10.355]
INFO:
prover-client:proving-broker Proving Broker started
12:10:10 [12:10:10.355]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f79eee8e0411966a epochNumber=1
{"provingJobId":"1:BASE_PARITY:f79eee8e0411966a"}
12:10:10 [12:10:10.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e713a9b98c4adae epochNumber=2
{"provingJobId":"1:BASE_PARITY:9e713a9b98c4adae"}
12:10:11 [12:10:11.357]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.357]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f79eee8e0411966a settled=false
{"provingJobId":"1:BASE_PARITY:f79eee8e0411966a","status":"pending"}
12:10:11 [12:10:11.357]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9e713a9b98c4adae settled=false
{"provingJobId":"1:BASE_PARITY:9e713a9b98c4adae","status":"pending"}
12:10:11 [12:10:11.358]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f79eee8e0411966a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f79eee8e0411966a"}
12:10:11 [12:10:11.358]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f79eee8e0411966a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f79eee8e0411966a"}
12:10:11 [12:10:11.359]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.360]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01c34bb265015bed epochNumber=1
{"provingJobId":"1:BASE_PARITY:01c34bb265015bed"}
12:10:11 [12:10:11.360]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fbf1dd306af08e01 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fbf1dd306af08e01"}
12:10:11 [12:10:11.361]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:01c34bb265015bed type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:01c34bb265015bed"}
12:10:11 [12:10:11.362]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fbf1dd306af08e01 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fbf1dd306af08e01"}
12:10:11 [12:10:11.364]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af7a840c7d8be4e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af7a840c7d8be4e8"}
12:10:11 [12:10:11.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2c19d89f118c217 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b2c19d89f118c217"}
12:10:11 [12:10:11.365]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:af7a840c7d8be4e8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:af7a840c7d8be4e8"}
12:10:11 [12:10:11.366]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:af7a840c7d8be4e8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:af7a840c7d8be4e8"}
12:10:11 [12:10:11.366]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b2c19d89f118c217 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b2c19d89f118c217"}
12:10:11 [12:10:11.367]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.368]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:29db6ad962895814 err=test error
{"provingJoId":"1:BASE_PARITY:29db6ad962895814"}
12:10:11 [12:10:11.369]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.369]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8730610d538b16b8 not found
{"provingJobId":"1:BASE_PARITY:8730610d538b16b8"}
12:10:11 [12:10:11.370]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.370]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc4bd47b653146d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc4bd47b653146d0"}
12:10:11 [12:10:11.372]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.372]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a7694327f9e89d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a7694327f9e89d1"}
12:10:11 [12:10:11.473]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8a7694327f9e89d1 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8a7694327f9e89d1"}
12:10:11 [12:10:11.482]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11af93805a987cb8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:11af93805a987cb8"}
12:10:11 [12:10:11.583]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:11af93805a987cb8 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:11af93805a987cb8"}
12:10:11 [12:10:11.590]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bde53f9bd4fa5a90 epochNumber=10
{"provingJobId":"1:BASE_PARITY:bde53f9bd4fa5a90"}
12:10:11 [12:10:11.610]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:11 [12:10:11.718]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.718]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:184dd9ffe32ce0db epochNumber=1
{"provingJobId":"1:BASE_PARITY:184dd9ffe32ce0db"}
12:10:11 [12:10:11.893]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:184dd9ffe32ce0db timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:184dd9ffe32ce0db"}
12:10:11 [12:10:11.927]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.928]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6009c6eb2a3c17d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6009c6eb2a3c17d1"}
12:10:11 [12:10:11.931]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6009c6eb2a3c17d1 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6009c6eb2a3c17d1"}
12:10:11 [12:10:11.933]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.934]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:124f05fe0a3df008 epochNumber=1
{"provingJobId":"1:BASE_PARITY:124f05fe0a3df008"}
12:10:11 [12:10:11.935]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:124f05fe0a3df008 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:124f05fe0a3df008"}
12:10:11 [12:10:11.936]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:124f05fe0a3df008 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:124f05fe0a3df008"}
12:10:11 [12:10:11.938]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58c5b6d5b2f6652f epochNumber=1
{"provingJobId":"1:BASE_PARITY:58c5b6d5b2f6652f"}
12:10:11 [12:10:11.941]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:58c5b6d5b2f6652f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:58c5b6d5b2f6652f"}
12:10:11 [12:10:11.942]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.943]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dab70a2b446e6366 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dab70a2b446e6366"}
12:10:11 [12:10:11.943]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:dab70a2b446e6366 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:dab70a2b446e6366"}
12:10:11 [12:10:11.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31c66641f684f113 epochNumber=3
{"provingJobId":"1:BASE_PARITY:31c66641f684f113"}
12:10:11 [12:10:11.968]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:11 [12:10:11.970]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:dab70a2b446e6366 err=test error
{"provingJoId":"1:BASE_PARITY:dab70a2b446e6366"}
12:10:11 [12:10:11.972]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.973]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9d116cfd8d51c8ee settled=false
{"provingJobId":"1:BASE_PARITY:9d116cfd8d51c8ee","status":"pending"}
12:10:11 [12:10:11.973]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f0169d5468deed12 settled=false
{"provingJobId":"1:BASE_PARITY:f0169d5468deed12","status":"pending"}
12:10:11 [12:10:11.975]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.976]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:539201f896ff8717 settled=true
{"provingJobId":"1:BASE_PARITY:539201f896ff8717","status":"fulfilled"}
12:10:11 [12:10:11.976]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:db680e1a1a1c89ff settled=true
{"provingJobId":"2:BASE_PARITY:db680e1a1a1c89ff","status":"fulfilled"}
12:10:11 [12:10:11.977]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.978]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:47ca3e87ecefea45 settled=true
{"provingJobId":"1:BASE_PARITY:47ca3e87ecefea45","status":"fulfilled"}
12:10:11 [12:10:11.978]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0bf4a9fbee6a179d settled=false
{"provingJobId":"1:BASE_PARITY:0bf4a9fbee6a179d","status":"pending"}
12:10:11 [12:10:11.979]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.980]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb9175995625461e epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb9175995625461e"}
12:10:11 [12:10:11.982]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.982]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d0c95304963f463 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1d0c95304963f463"}
12:10:11 [12:10:11.983]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:1d0c95304963f463: Error: db error: Error: db error
12:10:11 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:1d0c95304963f463"}
12:10:11 [12:10:11.984]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.984]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11715fd6fe46f898 epochNumber=1
{"provingJobId":"1:BASE_PARITY:11715fd6fe46f898"}
12:10:11 [12:10:11.985]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:11715fd6fe46f898 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:11715fd6fe46f898"}
12:10:11 [12:10:11.985]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:11715fd6fe46f898 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:11715fd6fe46f898"}
12:10:11 [12:10:11.986]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.987]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:63aa00fdfe1692ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:63aa00fdfe1692ad"}
12:10:11 [12:10:11.987]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:63aa00fdfe1692ad type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:63aa00fdfe1692ad"}
12:10:11 [12:10:11.987]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:63aa00fdfe1692ad type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:63aa00fdfe1692ad"}
12:10:11 [12:10:11.988]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:63aa00fdfe1692ad: Error: db error
12:10:11 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:63aa00fdfe1692ad"}
12:10:11 [12:10:11.992]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.992]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af09bcb6e34eed77 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af09bcb6e34eed77"}
12:10:11 [12:10:11.993]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:af09bcb6e34eed77 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:af09bcb6e34eed77"}
12:10:11 [12:10:11.993]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:af09bcb6e34eed77 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:af09bcb6e34eed77"}
12:10:11 [12:10:11.994]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.995]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:134b460513475f34 epochNumber=1
{"provingJobId":"1:BASE_PARITY:134b460513475f34"}
12:10:11 [12:10:11.995]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:134b460513475f34 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:134b460513475f34"}
12:10:11 [12:10:11.995]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:134b460513475f34 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:134b460513475f34"}
12:10:11 [12:10:11.996]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:134b460513475f34 jobErr=test error: Error: db error
12:10:11 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:134b460513475f34"}
12:10:11 [12:10:11.997]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.998]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:09138a47b662a5e4 not found
{"provingJobId":"1:BASE_PARITY:09138a47b662a5e4"}
12:10:11 [12:10:11.999]
INFO:
prover-client:proving-broker Proving Broker started
12:10:11 [12:10:11.999]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:5c23ceef3fde6ad6 err=test error
{"provingJoId":"1:BASE_PARITY:5c23ceef3fde6ad6"}
12:10:11 [12:10:12.000]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.011]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a72f47583c27da7a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a72f47583c27da7a"}
12:10:12 [12:10:12.011]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a72f47583c27da7a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a72f47583c27da7a"}
12:10:12 [12:10:12.011]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a72f47583c27da7a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a72f47583c27da7a"}
12:10:12 [12:10:12.024]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:2e70c0efeaa119be epochNumber=2
{"provingJobId":"2:BASE_PARITY:2e70c0efeaa119be"}
12:10:12 [12:10:12.024]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:2e70c0efeaa119be type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:2e70c0efeaa119be"}
12:10:12 [12:10:12.025]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2e70c0efeaa119be type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:2e70c0efeaa119be"}
12:10:12 [12:10:12.086]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:d11a718305b4faf6 epochNumber=3
{"provingJobId":"3:BASE_PARITY:d11a718305b4faf6"}
12:10:12 [12:10:12.104]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:12 [12:10:12.148]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:66466c79229efc78 epochNumber=4
{"provingJobId":"4:BASE_PARITY:66466c79229efc78"}
12:10:12 [12:10:12.155]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:12 [12:10:12.210]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:017f7b3a782daa63 epochNumber=5
{"provingJobId":"5:BASE_PARITY:017f7b3a782daa63"}
12:10:12 [12:10:12.231]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:12 [12:10:12.278]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44f009e2daeeadc9 epochNumber=42
{"provingJobId":"1:BASE_PARITY:44f009e2daeeadc9"}
12:10:12 [12:10:12.280]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testBqG3g7/42 with map size 1073741824
12:10:12 [12:10:12.286]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.326]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:1b36d263f1e15a03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b36d263f1e15a03"}
12:10:12 [12:10:12.334]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa09ba9c5c0f300a epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa09ba9c5c0f300a"}
12:10:12 [12:10:12.337]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCgN34e/1 with map size 1073741824
12:10:12 [12:10:12.340]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.370]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0e17b1a4a929982 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0e17b1a4a929982"}
12:10:12 [12:10:12.381]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.382]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58f8cc27f891f547 epochNumber=1
{"provingJobId":"1:BASE_PARITY:58f8cc27f891f547"}
12:10:12 [12:10:12.383]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testr5Wv1w/1 with map size 1073741824
12:10:12 [12:10:12.385]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.408]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:58f8cc27f891f547 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:58f8cc27f891f547"}
12:10:12 [12:10:12.412]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.413]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d5f0b589a99278e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d5f0b589a99278e3"}
12:10:12 [12:10:12.414]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPRvPkB/1 with map size 1073741824
12:10:12 [12:10:12.415]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.446]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d5f0b589a99278e3 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d5f0b589a99278e3"}
12:10:12 [12:10:12.447]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d5f0b589a99278e3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d5f0b589a99278e3"}
12:10:12 [12:10:12.455]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d5f0b589a99278e3 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d5f0b589a99278e3"}
12:10:12 [12:10:12.460]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.461]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5cef6734ab5bbf3c epochNumber=1
{"provingJobId":"1:BASE_PARITY:5cef6734ab5bbf3c"}
12:10:12 [12:10:12.462]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfI5DUb/1 with map size 1073741824
12:10:12 [12:10:12.464]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.490]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.495]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.496]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e87c4ed4de85a2ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:e87c4ed4de85a2ba"}
12:10:12 [12:10:12.497]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH49Y8V/1 with map size 1073741824
12:10:12 [12:10:12.500]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.520]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e87c4ed4de85a2ba
{"provingJobId":"1:BASE_PARITY:e87c4ed4de85a2ba"}
12:10:12 [12:10:12.522]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e87c4ed4de85a2ba type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e87c4ed4de85a2ba"}
12:10:12 [12:10:12.523]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e87c4ed4de85a2ba type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:e87c4ed4de85a2ba"}
12:10:12 [12:10:12.541]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.542]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:163ff4afd621f965 epochNumber=1
{"provingJobId":"1:BASE_PARITY:163ff4afd621f965"}
12:10:12 [12:10:12.543]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIax4WM/1 with map size 1073741824
12:10:12 [12:10:12.545]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.565]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:163ff4afd621f965
{"provingJobId":"1:BASE_PARITY:163ff4afd621f965"}
12:10:12 [12:10:12.566]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:163ff4afd621f965 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:163ff4afd621f965"}
12:10:12 [12:10:12.579]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.580]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:358ec58fbb038164 epochNumber=1
{"provingJobId":"1:BASE_PARITY:358ec58fbb038164"}
12:10:12 [12:10:12.580]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHv19I4/1 with map size 1073741824
12:10:12 [12:10:12.583]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.599]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:358ec58fbb038164 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:358ec58fbb038164"}
12:10:12 [12:10:12.600]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:358ec58fbb038164 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:358ec58fbb038164"}
12:10:12 [12:10:12.609]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.610]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:426f52d816ca0591 epochNumber=1
{"provingJobId":"1:BASE_PARITY:426f52d816ca0591"}
12:10:12 [12:10:12.611]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzNh17Q/1 with map size 1073741824
12:10:12 [12:10:12.616]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.642]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:426f52d816ca0591 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:426f52d816ca0591"}
12:10:12 [12:10:12.643]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:426f52d816ca0591 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:426f52d816ca0591"}
12:10:12 [12:10:12.655]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.655]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3d814bb6ce548f7 epochNumber=0
{"provingJobId":"1:BASE_PARITY:f3d814bb6ce548f7"}
12:10:12 [12:10:12.656]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testeOuRSx/0 with map size 1073741824
12:10:12 [12:10:12.658]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.675]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f3d814bb6ce548f7 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f3d814bb6ce548f7"}
12:10:12 [12:10:12.676]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2cc300554de4adf8 epochNumber=0
{"provingJobId":"1:BASE_PARITY:2cc300554de4adf8"}
12:10:12 [12:10:12.676]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f3d814bb6ce548f7 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f3d814bb6ce548f7"}
12:10:12 [12:10:12.677]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de32c890d48f0126 epochNumber=0
{"provingJobId":"1:BASE_PARITY:de32c890d48f0126"}
12:10:12 [12:10:12.692]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.694]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.695]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec7156fd9001ce22 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ec7156fd9001ce22"}
12:10:12 [12:10:12.695]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcpjvav/2 with map size 1073741824
12:10:12 [12:10:12.697]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.715]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c226b4c2f8a0aa3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c226b4c2f8a0aa3"}
12:10:12 [12:10:12.716]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcpjvav/1 with map size 1073741824
12:10:12 [12:10:12.719]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.750]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.751]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7fe0ab55d4039b84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7fe0ab55d4039b84"}
12:10:12 [12:10:12.752]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testU6w70V/1 with map size 1073741824
12:10:12 [12:10:12.754]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.774]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.774]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1cc58f8add6185a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1cc58f8add6185a6"}
12:10:12 [12:10:12.775]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxanoCW/1 with map size 1073741824
12:10:12 [12:10:12.777]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.795]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c80bc5897bdbef43 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c80bc5897bdbef43"}
12:10:12 [12:10:12.803]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9540311197033cd0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9540311197033cd0"}
12:10:12 [12:10:12.804]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testxanoCW/2 with map size 1073741824
12:10:12 [12:10:12.806]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.829]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59c2ca07dfba5d35 epochNumber=1
{"provingJobId":"1:BASE_PARITY:59c2ca07dfba5d35"}
12:10:12 [12:10:12.844]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.845]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc38307552ee235f epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc38307552ee235f"}
12:10:12 [12:10:12.845]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPm4ju2/1 with map size 1073741824
12:10:12 [12:10:12.848]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.868]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5484b2dd003d63e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5484b2dd003d63e9"}
12:10:12 [12:10:12.875]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4adc3b3e9eec6151 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4adc3b3e9eec6151"}
12:10:12 [12:10:12.876]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testPm4ju2/2 with map size 1073741824
12:10:12 [12:10:12.879]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.897]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1989883a3ed54e38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1989883a3ed54e38"}
12:10:12 [12:10:12.908]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.909]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:314e60d1f8718388 epochNumber=1
{"provingJobId":"1:BASE_PARITY:314e60d1f8718388"}
12:10:12 [12:10:12.909]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcaLrmI/1 with map size 1073741824
12:10:12 [12:10:12.912]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.934]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:712235cf2b1e6497 epochNumber=1
{"provingJobId":"1:BASE_PARITY:712235cf2b1e6497"}
12:10:12 [12:10:12.948]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f20984c2df2cd9f5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f20984c2df2cd9f5"}
12:10:12 [12:10:12.949]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcaLrmI/2 with map size 1073741824
12:10:12 [12:10:12.952]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:12 [12:10:12.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b423a160746596d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b423a160746596d2"}
12:10:12 [12:10:12.983]
INFO:
prover-client:proving-broker Proving Broker started
12:10:12 [12:10:12.984]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3e347b72787509d epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3e347b72787509d"}
12:10:12 [12:10:12.984]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKMdWea/1 with map size 1073741824
12:10:12 [12:10:12.987]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:13 [12:10:13.005]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:278c55217258598a epochNumber=1
{"provingJobId":"1:BASE_PARITY:278c55217258598a"}
12:10:13 [12:10:13.012]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a3e347b72787509d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a3e347b72787509d"}
12:10:13 [12:10:13.029]
INFO:
prover-client:proving-broker Proving Broker started
12:10:13 [12:10:13.031]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4ad383937f69868 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4ad383937f69868"}
12:10:13 [12:10:13.033]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testO1vBAC/1 with map size 1073741824
12:10:13 [12:10:13.035]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:13 [12:10:13.055]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dff2b7add3ec30f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dff2b7add3ec30f5"}
12:10:13 [12:10:13.061]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f4ad383937f69868 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:f4ad383937f69868"}
12:10:13 [12:10:13.072]
INFO:
prover-client:proving-broker Proving Broker started
12:10:13 [12:10:13.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f494a3a94a480e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f494a3a94a480e9"}
12:10:13 [12:10:13.073]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFl2gKi/1 with map size 1073741824
12:10:13 [12:10:13.075]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:13 [12:10:13.093]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f407bc5fd8457d09 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f407bc5fd8457d09"}
12:10:13 [12:10:13.100]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1f494a3a94a480e9 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:1f494a3a94a480e9"}
12:10:13 [12:10:13.104]
INFO:
prover-client:proving-broker Proving Broker started
12:10:13 [12:10:13.106]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:106f3f2fd5633c76 epochNumber=1
{"provingJobId":"1:BASE_PARITY:106f3f2fd5633c76"}
12:10:13 [12:10:13.107]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCMw96J/1 with map size 1073741824
12:10:13 [12:10:13.109]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:13 [12:10:13.130]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:106f3f2fd5633c76
{"provingJobId":"1:BASE_PARITY:106f3f2fd5633c76"}
12:10:13 [12:10:13.131]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:106f3f2fd5633c76 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:106f3f2fd5633c76"}
12:10:13 [12:10:13.143]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b62c4e418456baf3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b62c4e418456baf3"}
12:10:13 [12:10:13.150]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:106f3f2fd5633c76 has already been completed
{"provingJobId":"1:BASE_PARITY:106f3f2fd5633c76"}
12:10:13 [12:10:13.155]
INFO:
prover-client:proving-broker Proving Broker started
12:10:13 [12:10:13.155]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c4ed089f31f7a47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c4ed089f31f7a47"}
12:10:13 [12:10:13.156]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9SiH4y/1 with map size 1073741824
12:10:13 [12:10:13.158]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:13 [12:10:13.175]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9d9438a0dee1481 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b9d9438a0dee1481"}
12:10:13 [12:10:13.176]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test9SiH4y/2 with map size 1073741824
12:10:13 [12:10:13.179]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:14 [12:10:14.247]
INFO:
prover-client:proving-broker Proving Broker started
12:10:14 [12:10:14.254]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9c4ed089f31f7a47 settled=false
{"provingJobId":"1:BASE_PARITY:9c4ed089f31f7a47","status":"pending"}
12:10:14 [12:10:14.256]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b9d9438a0dee1481 settled=false
{"provingJobId":"1:BASE_PARITY:b9d9438a0dee1481","status":"pending"}
12:10:14 [12:10:14.257]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9c4ed089f31f7a47 type=BASE_PARITY startedAt=1749730213195 older agent has taken job
{"provingJobId":"1:BASE_PARITY:9c4ed089f31f7a47"}
12:10:14 [12:10:14.258]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9c4ed089f31f7a47 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:9c4ed089f31f7a47"}
12:10:14 [12:10:14.261]
INFO:
prover-client:proving-broker Proving Broker started
12:10:14 [12:10:14.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b867522509e45a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b867522509e45a0"}
12:10:14 [12:10:14.262]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYA4NrX/1 with map size 1073741824
12:10:14 [12:10:14.264]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:14 [12:10:14.283]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:812684096d373b4f epochNumber=2
{"provingJobId":"1:BASE_PARITY:812684096d373b4f"}
12:10:14 [12:10:14.284]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYA4NrX/2 with map size 1073741824
12:10:14 [12:10:14.287]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:15 [12:10:15.306]
INFO:
prover-client:proving-broker Proving Broker started
12:10:15 [12:10:15.308]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6b867522509e45a0 settled=false
{"provingJobId":"1:BASE_PARITY:6b867522509e45a0","status":"pending"}
12:10:15 [12:10:15.310]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:812684096d373b4f settled=false
{"provingJobId":"1:BASE_PARITY:812684096d373b4f","status":"pending"}
12:10:15 [12:10:15.310]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6b867522509e45a0 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:6b867522509e45a0"}
12:10:15 [12:10:15.315]
INFO:
prover-client:proving-broker Proving Broker started
12:10:15 [12:10:15.316]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01b47458a5a14a95 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01b47458a5a14a95"}
12:10:15 [12:10:15.317]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0NqAMf/1 with map size 1073741824
12:10:15 [12:10:15.319]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:15 [12:10:15.342]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08dc2855380a7f1b epochNumber=2
{"provingJobId":"1:BASE_PARITY:08dc2855380a7f1b"}
12:10:15 [12:10:15.344]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0NqAMf/2 with map size 1073741824
12:10:15 [12:10:15.346]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:16 [12:10:16.366]
INFO:
prover-client:proving-broker Proving Broker started
12:10:16 [12:10:16.367]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:01b47458a5a14a95 settled=false
{"provingJobId":"1:BASE_PARITY:01b47458a5a14a95","status":"pending"}
12:10:16 [12:10:16.369]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:08dc2855380a7f1b settled=false
{"provingJobId":"1:BASE_PARITY:08dc2855380a7f1b","status":"pending"}
12:10:16 [12:10:16.370]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:01b47458a5a14a95 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:01b47458a5a14a95"}
12:10:16 [12:10:16.370]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:01b47458a5a14a95 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:01b47458a5a14a95"}
12:10:16 [12:10:16.380]
INFO:
prover-client:proving-broker Proving Broker started
12:10:16 [12:10:16.381]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:599445b7e4d1bc6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:599445b7e4d1bc6a"}
12:10:16 [12:10:16.382]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLkXrji/1 with map size 1073741824
12:10:16 [12:10:16.385]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:16 [12:10:16.400]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1535e17e548957ba epochNumber=2
{"provingJobId":"1:BASE_PARITY:1535e17e548957ba"}
12:10:16 [12:10:16.401]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testLkXrji/2 with map size 1073741824
12:10:16 [12:10:16.403]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:16 [12:10:16.422]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:599445b7e4d1bc6a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:599445b7e4d1bc6a"}
12:10:16 [12:10:16.430]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1535e17e548957ba type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1535e17e548957ba"}
12:10:16 [12:10:16.443]
INFO:
prover-client:proving-broker Proving Broker started
12:10:16 [12:10:16.444]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0329a1a49d74223a epochNumber=1
{"provingJobId":"1:BASE_PARITY:0329a1a49d74223a"}
12:10:16 [12:10:16.444]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtZu0Bk/1 with map size 1073741824
12:10:16 [12:10:16.447]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:16 [12:10:16.467]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a9f8104cb227306 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9a9f8104cb227306"}
12:10:16 [12:10:16.467]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtZu0Bk/2 with map size 1073741824
12:10:16 [12:10:16.471]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:16 [12:10:16.490]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0329a1a49d74223a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0329a1a49d74223a"}
12:10:16 [12:10:16.490]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0329a1a49d74223a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0329a1a49d74223a"}
12:10:16 [12:10:16.498]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9a9f8104cb227306 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9a9f8104cb227306"}
12:10:16 [12:10:16.508]
INFO:
prover-client:proving-broker Proving Broker started
12:10:16 [12:10:16.509]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d0e9fd4c4d7871fa err=test error
{"provingJoId":"1:BASE_PARITY:d0e9fd4c4d7871fa"}
12:10:16 [12:10:16.510]
INFO:
prover-client:proving-broker Proving Broker started
12:10:16 [12:10:16.511]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:59f74a86fe47d670 not found
{"provingJobId":"1:BASE_PARITY:59f74a86fe47d670"}
12:10:16 [12:10:16.513]
INFO:
prover-client:proving-broker Proving Broker started
12:10:16 [12:10:16.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72ebe565b5cc2d14 epochNumber=1
{"provingJobId":"1:BASE_PARITY:72ebe565b5cc2d14"}
12:10:16 [12:10:16.515]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhInuPq/1 with map size 1073741824
12:10:16 [12:10:16.516]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:16 [12:10:16.540]
INFO:
prover-client:proving-broker Proving Broker started
12:10:16 [12:10:16.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6a962406e84f1c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d6a962406e84f1c7"}
12:10:16 [12:10:16.542]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHjbzRg/1 with map size 1073741824
12:10:16 [12:10:16.544]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:16 [12:10:16.667]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d6a962406e84f1c7 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:d6a962406e84f1c7"}
12:10:16 [12:10:16.672]
INFO:
prover-client:proving-broker Proving Broker started
12:10:16 [12:10:16.673]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:709c7f79a5c257dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:709c7f79a5c257dd"}
12:10:16 [12:10:16.673]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testN8BOLU/1 with map size 1073741824
12:10:16 [12:10:16.676]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:16 [12:10:16.799]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:709c7f79a5c257dd timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:709c7f79a5c257dd"}
12:10:16 [12:10:16.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57989340f63200d9 epochNumber=10
{"provingJobId":"1:BASE_PARITY:57989340f63200d9"}
12:10:16 [12:10:16.805]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testN8BOLU/10 with map size 1073741824
12:10:16 [12:10:16.809]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:16 [12:10:16.850]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:16 [12:10:16.850]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:10:16 [12:10:16.961]
INFO:
prover-client:proving-broker Proving Broker started
12:10:16 [12:10:16.961]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:859473cf715ae490 epochNumber=1
{"provingJobId":"1:BASE_PARITY:859473cf715ae490"}
12:10:16 [12:10:16.962]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcFSx8L/1 with map size 1073741824
12:10:16 [12:10:16.963]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.163]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:859473cf715ae490 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:859473cf715ae490"}
12:10:17 [12:10:17.194]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.195]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d66820b76dc67533 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d66820b76dc67533"}
12:10:17 [12:10:17.195]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBAeo4F/1 with map size 1073741824
12:10:17 [12:10:17.197]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.212]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d66820b76dc67533 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d66820b76dc67533"}
12:10:17 [12:10:17.215]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.215]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5aae39a43bd6521 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5aae39a43bd6521"}
12:10:17 [12:10:17.216]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testO5mqZt/1 with map size 1073741824
12:10:17 [12:10:17.217]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.232]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b5aae39a43bd6521 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b5aae39a43bd6521"}
12:10:17 [12:10:17.232]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b5aae39a43bd6521 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:b5aae39a43bd6521"}
12:10:17 [12:10:17.241]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.241]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:009a16b9897c62b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:009a16b9897c62b3"}
12:10:17 [12:10:17.242]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaRd5sc/1 with map size 1073741824
12:10:17 [12:10:17.243]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.257]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:009a16b9897c62b3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:009a16b9897c62b3"}
12:10:17 [12:10:17.265]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c15513030535d8f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c15513030535d8f"}
12:10:17 [12:10:17.266]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpKzLDq/1 with map size 1073741824
12:10:17 [12:10:17.267]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.285]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4c15513030535d8f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:4c15513030535d8f"}
12:10:17 [12:10:17.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d15c24e8c050a296 epochNumber=3
{"provingJobId":"1:BASE_PARITY:d15c24e8c050a296"}
12:10:17 [12:10:17.286]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testpKzLDq/3 with map size 1073741824
12:10:17 [12:10:17.288]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.316]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:17 [12:10:17.316]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:10:17 [12:10:17.332]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4c15513030535d8f err=test error
{"provingJoId":"1:BASE_PARITY:4c15513030535d8f"}
12:10:17 [12:10:17.338]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnpChPu/1 with map size 1073741824
12:10:17 [12:10:17.340]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.363]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnpChPu/2 with map size 1073741824
12:10:17 [12:10:17.366]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.384]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.386]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ad3f62869afe97dd settled=false
{"provingJobId":"1:BASE_PARITY:ad3f62869afe97dd","status":"pending"}
12:10:17 [12:10:17.387]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a044e07af6d72ce6 settled=false
{"provingJobId":"1:BASE_PARITY:a044e07af6d72ce6","status":"pending"}
12:10:17 [12:10:17.392]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZRknad/1 with map size 1073741824
12:10:17 [12:10:17.394]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.412]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZRknad/2 with map size 1073741824
12:10:17 [12:10:17.415]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.446]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.450]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:87959402d9c92794 settled=true
{"provingJobId":"1:BASE_PARITY:87959402d9c92794","status":"fulfilled"}
12:10:17 [12:10:17.452]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b3e9a4fa3924e971 settled=true
{"provingJobId":"2:BASE_PARITY:b3e9a4fa3924e971","status":"fulfilled"}
12:10:17 [12:10:17.459]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAO7PMf/1 with map size 1073741824
12:10:17 [12:10:17.461]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.489]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAO7PMf/2 with map size 1073741824
12:10:17 [12:10:17.492]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.515]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.517]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b6a6d482c20f4988 settled=true
{"provingJobId":"1:BASE_PARITY:b6a6d482c20f4988","status":"fulfilled"}
12:10:17 [12:10:17.520]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:20f699c3e0581b5f settled=false
{"provingJobId":"1:BASE_PARITY:20f699c3e0581b5f","status":"pending"}
12:10:17 [12:10:17.527]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.528]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7844c979c5e17467 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7844c979c5e17467"}
12:10:17 [12:10:17.529]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiuQcDa/1 with map size 1073741824
12:10:17 [12:10:17.532]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.557]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.558]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1cea59d48f66a634 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1cea59d48f66a634"}
12:10:17 [12:10:17.558]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:1cea59d48f66a634: Error: db error: Error: db error
12:10:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:1cea59d48f66a634"}
12:10:17 [12:10:17.564]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.564]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13175f2749e527b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13175f2749e527b6"}
12:10:17 [12:10:17.566]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOlRbnC/1 with map size 1073741824
12:10:17 [12:10:17.569]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.587]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:13175f2749e527b6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:13175f2749e527b6"}
12:10:17 [12:10:17.588]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:13175f2749e527b6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:13175f2749e527b6"}
12:10:17 [12:10:17.601]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.601]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:516842fb3df77888 epochNumber=1
{"provingJobId":"1:BASE_PARITY:516842fb3df77888"}
12:10:17 [12:10:17.602]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQ4KOC4/1 with map size 1073741824
12:10:17 [12:10:17.606]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.625]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:516842fb3df77888 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:516842fb3df77888"}
12:10:17 [12:10:17.626]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:516842fb3df77888 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:516842fb3df77888"}
12:10:17 [12:10:17.627]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:516842fb3df77888: Error: db error
12:10:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:516842fb3df77888"}
12:10:17 [12:10:17.634]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.635]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a765ca3a20de2e50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a765ca3a20de2e50"}
12:10:17 [12:10:17.636]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZth2kD/1 with map size 1073741824
12:10:17 [12:10:17.639]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.660]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a765ca3a20de2e50 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a765ca3a20de2e50"}
12:10:17 [12:10:17.661]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a765ca3a20de2e50 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a765ca3a20de2e50"}
12:10:17 [12:10:17.671]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.671]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80539049c801a09c epochNumber=1
{"provingJobId":"1:BASE_PARITY:80539049c801a09c"}
12:10:17 [12:10:17.672]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwF3ScA/1 with map size 1073741824
12:10:17 [12:10:17.675]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.693]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:80539049c801a09c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:80539049c801a09c"}
12:10:17 [12:10:17.694]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:80539049c801a09c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:80539049c801a09c"}
12:10:17 [12:10:17.695]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:80539049c801a09c jobErr=test error: Error: db error
12:10:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:80539049c801a09c"}
12:10:17 [12:10:17.700]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.700]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b79e12ee99e1988 not found
{"provingJobId":"1:BASE_PARITY:7b79e12ee99e1988"}
12:10:17 [12:10:17.705]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.706]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7de33bed6b0dd831 err=test error
{"provingJoId":"1:BASE_PARITY:7de33bed6b0dd831"}
12:10:17 [12:10:17.715]
INFO:
prover-client:proving-broker Proving Broker started
12:10:17 [12:10:17.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91aa65ac3292969c epochNumber=1
{"provingJobId":"1:BASE_PARITY:91aa65ac3292969c"}
12:10:17 [12:10:17.728]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiF1mIN/1 with map size 1073741824
12:10:17 [12:10:17.731]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.752]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:91aa65ac3292969c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:91aa65ac3292969c"}
12:10:17 [12:10:17.752]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:91aa65ac3292969c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:91aa65ac3292969c"}
12:10:17 [12:10:17.769]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:63c782fac46ee354 epochNumber=2
{"provingJobId":"2:BASE_PARITY:63c782fac46ee354"}
12:10:17 [12:10:17.770]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testiF1mIN/2 with map size 1073741824
12:10:17 [12:10:17.774]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.794]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:63c782fac46ee354 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:63c782fac46ee354"}
12:10:17 [12:10:17.794]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:63c782fac46ee354 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:63c782fac46ee354"}
12:10:17 [12:10:17.862]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:ad6ac7eb271a19d9 epochNumber=3
{"provingJobId":"3:BASE_PARITY:ad6ac7eb271a19d9"}
12:10:17 [12:10:17.864]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testiF1mIN/3 with map size 1073741824
12:10:17 [12:10:17.866]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.895]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:17 [12:10:17.895]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:10:17 [12:10:17.952]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:e03664d01aaff899 epochNumber=4
{"provingJobId":"4:BASE_PARITY:e03664d01aaff899"}
12:10:17 [12:10:17.953]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testiF1mIN/4 with map size 1073741824
12:10:17 [12:10:17.956]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:17 [12:10:17.974]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:17 [12:10:17.975]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
12:10:18 [12:10:18.033]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:75565482ad1623fb epochNumber=5
{"provingJobId":"5:BASE_PARITY:75565482ad1623fb"}
12:10:18 [12:10:18.034]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testiF1mIN/5 with map size 1073741824
12:10:18 [12:10:18.037]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:10:18 [12:10:18.079]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:10:18 [12:10:18.079]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
12:10:18
PASS src/proving_broker/proving_broker.test.ts (
12.496 s)
12:10:18 ProvingBroker
12:10:18 Producer API
12:10:18
✓ refuses stale jobs (39 ms)
12:10:18
✓ enqueues jobs (3 ms)
12:10:18
✓ ignores duplicate jobs (3 ms)
12:10:18
✓ reports correct status when enqueuing repeat jobs (4 ms)
12:10:18
✓ throws an error in case of duplicate job IDs (28 ms)
12:10:18
✓ returns not-found status for non-existing jobs (2 ms)
12:10:18
✓ cancels jobs in queue (3 ms)
12:10:18
✓ cancels jobs in-progress (1 ms)
12:10:18
✓ returns job result if successful (1 ms)
12:10:18
✓ returns job error if failed (1 ms)
12:10:18
✓ correctly returns job status for concurrent writes (2 ms)
12:10:18
✓ refuses stale jobs (58 ms)
12:10:18
✓ enqueues jobs (46 ms)
12:10:18
✓ ignores duplicate jobs (31 ms)
12:10:18
✓ reports correct status when enqueuing repeat jobs (46 ms)
12:10:18
✓ throws an error in case of duplicate job IDs (29 ms)
12:10:18
✓ returns not-found status for non-existing jobs (6 ms)
12:10:18
✓ cancels jobs in queue (45 ms)
12:10:18
✓ cancels jobs in-progress (37 ms)
12:10:18
✓ returns job result if successful (30 ms)
12:10:18
✓ returns job error if failed (44 ms)
12:10:18
✓ correctly returns job status for concurrent writes (39 ms)
12:10:18 Consumer API
12:10:18
✓ returns undefined if no jobs are available (1 ms)
12:10:18
✓ returns jobs in priority order (1 ms)
12:10:18
✓ returns undefined if no jobs are available for the given allowList (1 ms)
12:10:18
✓ returns a job if it is in the allowList (2 ms)
12:10:18
✓ returns the most important job if it is in the allowList (2 ms)
12:10:18
✓ returns any job if filter is empty (12 ms)
12:10:18
✓ returns a new job when reporting job success (2 ms)
12:10:18
✓ returns a new job when reporting permanent error (2 ms)
12:10:18
✓ returns a new job when reporting retry-able error (3 ms)
12:10:18
✓ returns a new job when reporting progress if current one is cancelled (4 ms)
12:10:18
✓ returns a new job if job is already in progress elsewhere (1056 ms)
12:10:18
✓ avoids sending the same job to a new agent after a restart (1004 ms)
12:10:18
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
12:10:18
✓ tracks job result if in progress (4 ms)
12:10:18
✓ tracks job result even if job is in queue (4 ms)
12:10:18
✓ ignores reported job error if unknown job (1 ms)
12:10:18
✓ ignores job result if unknown job (1 ms)
12:10:18
✓ returns undefined if no jobs are available (2 ms)
12:10:18
✓ returns jobs in priority order (54 ms)
12:10:18
✓ returns undefined if no jobs are available for the given allowList (24 ms)
12:10:18
✓ returns a job if it is in the allowList (69 ms)
12:10:18
✓ returns the most important job if it is in the allowList (65 ms)
12:10:18
✓ returns any job if filter is empty (74 ms)
12:10:18
✓ returns a new job when reporting job success (39 ms)
12:10:18
✓ returns a new job when reporting permanent error (49 ms)
12:10:18
✓ returns a new job when reporting retry-able error (32 ms)
12:10:18
✓ returns a new job when reporting progress if current one is cancelled (50 ms)
12:10:18
✓ returns a new job if job is already in progress elsewhere (1106 ms)
12:10:18
✓ avoids sending the same job to a new agent after a restart (1054 ms)
12:10:18
✓ avoids sending a completed job to a new agent after a restart (1065 ms)
12:10:18
✓ tracks job result if in progress (62 ms)
12:10:18
✓ tracks job result even if job is in queue (64 ms)
12:10:18
✓ ignores reported job error if unknown job (4 ms)
12:10:18
✓ ignores job result if unknown job (3 ms)
12:10:18 Timeouts
12:10:18
✓ tracks in progress jobs (1 ms)
12:10:18
✓ re-enqueues jobs that time out (110 ms)
12:10:18
✓ cancel stale jobs that time out (236 ms)
12:10:18
✓ keeps the jobs in progress while it is alive (208 ms)
12:10:18
✓ tracks in progress jobs (26 ms)
12:10:18
✓ re-enqueues jobs that time out (131 ms)
12:10:18
✓ cancel stale jobs that time out (290 ms)
12:10:18
✓ keeps the jobs in progress while it is alive (233 ms)
12:10:18 Retries
12:10:18
✓ retries jobs (6 ms)
12:10:18
✓ retries up to a maximum number of times (4 ms)
12:10:18
✓ passing retry=false does not retry (4 ms)
12:10:18
✓ does not retry if job is stale (29 ms)
12:10:18
✓ retries jobs (20 ms)
12:10:18
✓ retries up to a maximum number of times (25 ms)
12:10:18
✓ passing retry=false does not retry (23 ms)
12:10:18
✓ does not retry if job is stale (70 ms)
12:10:18 Database management
12:10:18
✓ re-enqueues proof requests on start (3 ms)
12:10:18
✓ restores proof results on start (2 ms)
12:10:18
✓ only re-enqueues unfinished jobs (1 ms)
12:10:18
✓ saves job when enqueued (3 ms)
12:10:18
✓ does not retain job if database fails to save (2 ms)
12:10:18
✓ saves job result (2 ms)
12:10:18
✓ saves result even if database fails to save (2 ms)
12:10:18
✓ saves job error (6 ms)
12:10:18
✓ saves job error even if database fails to save (2 ms)
12:10:18
✓ does not save job result if job is unknown (2 ms)
12:10:18
✓ does not save job error if job is unknown (1 ms)
12:10:18
✓ cleans up old jobs periodically (271 ms)
12:10:18
✓ re-enqueues proof requests on start (56 ms)
12:10:18
✓ restores proof results on start (66 ms)
12:10:18
✓ only re-enqueues unfinished jobs (67 ms)
12:10:18
✓ saves job when enqueued (31 ms)
12:10:18
✓ does not retain job if database fails to save (5 ms)
12:10:18
✓ saves job result (38 ms)
12:10:18
✓ saves result even if database fails to save (30 ms)
12:10:18
✓ saves job error (39 ms)
12:10:18
✓ saves job error even if database fails to save (28 ms)
12:10:18
✓ does not save job result if job is unknown (5 ms)
12:10:18
✓ does not save job error if job is unknown (7 ms)
12:10:18
✓ cleans up old jobs periodically (398 ms)
12:10:18
12:10:18
Test Suites: 1 passed, 1 total
12:10:18
Tests: 96 passed, 96 total
12:10:18
Snapshots: 0 total
12:10:18
Time: 12.582 s
12:10:18
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
12:10:18
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?