Parent Log:
http://ci.aztec-labs.com/96e3fbaf87d764c6
Command: bdb887914394d73a yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/47c749527b1c3cce3edd9defedc46e89ea00c69e
Env: REF_NAME=gh-readonly-queue/next/pr-15009-00cd67e98e76d3202a0219baf301d59ef4211872 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 13 15:54:20 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15009_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
15:54:24 [15:54:24.180]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.186]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c94eeb469f75e6c1 epochNumber=42
{"provingJobId":"1:BASE_PARITY:c94eeb469f75e6c1"}
15:54:24 [15:54:24.190]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:659cf703d4069e63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:659cf703d4069e63"}
15:54:24 [15:54:24.215]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.216]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6ed2e9cc655b559 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b6ed2e9cc655b559"}
15:54:24 [15:54:24.218]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47813f22ed362a7b epochNumber=1
{"provingJobId":"1:BASE_PARITY:47813f22ed362a7b"}
15:54:24 [15:54:24.220]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.221]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ea14349499d3fda epochNumber=1
{"provingJobId":"1:BASE_PARITY:0ea14349499d3fda"}
15:54:24 [15:54:24.222]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0ea14349499d3fda epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0ea14349499d3fda"}
15:54:24 [15:54:24.223]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.224]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:768927db02369f47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:768927db02369f47"}
15:54:24 [15:54:24.225]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:768927db02369f47 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:768927db02369f47"}
15:54:24 [15:54:24.226]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:768927db02369f47 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:768927db02369f47"}
15:54:24 [15:54:24.226]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:768927db02369f47 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:768927db02369f47"}
15:54:24 [15:54:24.227]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.228]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7785660ef1c6c979 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7785660ef1c6c979"}
15:54:24 [15:54:24.249]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.251]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.251]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5deeefa62ccb943d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5deeefa62ccb943d"}
15:54:24 [15:54:24.252]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5deeefa62ccb943d
{"provingJobId":"1:BASE_PARITY:5deeefa62ccb943d"}
15:54:24 [15:54:24.252]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5deeefa62ccb943d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5deeefa62ccb943d"}
15:54:24 [15:54:24.252]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5deeefa62ccb943d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5deeefa62ccb943d"}
15:54:24 [15:54:24.253]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.254]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d771de2b46a8228 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d771de2b46a8228"}
15:54:24 [15:54:24.254]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6d771de2b46a8228
{"provingJobId":"1:BASE_PARITY:6d771de2b46a8228"}
15:54:24 [15:54:24.254]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6d771de2b46a8228 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:6d771de2b46a8228"}
15:54:24 [15:54:24.256]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e04bae6748b9ee15 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e04bae6748b9ee15"}
15:54:24 [15:54:24.257]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e04bae6748b9ee15 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e04bae6748b9ee15"}
15:54:24 [15:54:24.257]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e04bae6748b9ee15 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e04bae6748b9ee15"}
15:54:24 [15:54:24.265]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13dd892478f7fa49 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13dd892478f7fa49"}
15:54:24 [15:54:24.266]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:13dd892478f7fa49 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:13dd892478f7fa49"}
15:54:24 [15:54:24.266]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:13dd892478f7fa49 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:13dd892478f7fa49"}
15:54:24 [15:54:24.267]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.267]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:325c93022c0fc15d epochNumber=0
{"provingJobId":"1:BASE_PARITY:325c93022c0fc15d"}
15:54:24 [15:54:24.268]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:325c93022c0fc15d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:325c93022c0fc15d"}
15:54:24 [15:54:24.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:efdc69c5dba60a6b epochNumber=0
{"provingJobId":"1:BASE_PARITY:efdc69c5dba60a6b"}
15:54:24 [15:54:24.268]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:325c93022c0fc15d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:325c93022c0fc15d"}
15:54:24 [15:54:24.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4cbf561e6eef6bb9 epochNumber=0
{"provingJobId":"1:BASE_PARITY:4cbf561e6eef6bb9"}
15:54:24 [15:54:24.269]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.270]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.270]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c39224f0ab3035ac epochNumber=2
{"provingJobId":"1:BASE_PARITY:c39224f0ab3035ac"}
15:54:24 [15:54:24.271]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0391a86f432cff51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0391a86f432cff51"}
15:54:24 [15:54:24.272]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.272]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:309e98acf03a331a epochNumber=1
{"provingJobId":"1:BASE_PARITY:309e98acf03a331a"}
15:54:24 [15:54:24.273]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a7d69d8b15e5fd7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a7d69d8b15e5fd7"}
15:54:24 [15:54:24.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79bc4e5b1f67dad9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:79bc4e5b1f67dad9"}
15:54:24 [15:54:24.274]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0d2524546209e0e epochNumber=2
{"provingJobId":"1:BASE_PARITY:f0d2524546209e0e"}
15:54:24 [15:54:24.274]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a0f88c8d2844568d epochNumber=1
{"provingJobId":"1:BASE_PARITY:a0f88c8d2844568d"}
15:54:24 [15:54:24.275]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.275]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:930426869349e058 epochNumber=1
{"provingJobId":"1:BASE_PARITY:930426869349e058"}
15:54:24 [15:54:24.276]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:384fc1eae0311239 epochNumber=1
{"provingJobId":"1:BASE_PARITY:384fc1eae0311239"}
15:54:24 [15:54:24.276]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6ce3fd0b12fd6ba epochNumber=2
{"provingJobId":"1:BASE_PARITY:c6ce3fd0b12fd6ba"}
15:54:24 [15:54:24.276]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef5d0b75ea6b759e epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef5d0b75ea6b759e"}
15:54:24 [15:54:24.277]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11cc3e372b803319 epochNumber=1
{"provingJobId":"1:BASE_PARITY:11cc3e372b803319"}
15:54:24 [15:54:24.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5107f855b04499cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:5107f855b04499cd"}
15:54:24 [15:54:24.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b110aaa9352ae11e epochNumber=2
{"provingJobId":"1:BASE_PARITY:b110aaa9352ae11e"}
15:54:24 [15:54:24.278]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d8c311b4026a8ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:2d8c311b4026a8ac"}
15:54:24 [15:54:24.278]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3d4721178b212e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3d4721178b212e7"}
15:54:24 [15:54:24.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56b29ef0db6685b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56b29ef0db6685b1"}
15:54:24 [15:54:24.279]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d3d4721178b212e7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d3d4721178b212e7"}
15:54:24 [15:54:24.281]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f9e419a9cec58bdd epochNumber=1
{"provingJobId":"1:BASE_PARITY:f9e419a9cec58bdd"}
15:54:24 [15:54:24.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42b8b19815ff1e8e epochNumber=1
{"provingJobId":"1:BASE_PARITY:42b8b19815ff1e8e"}
15:54:24 [15:54:24.281]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f9e419a9cec58bdd type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:f9e419a9cec58bdd"}
15:54:24 [15:54:24.282]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f4a7e9689edc8ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f4a7e9689edc8ed"}
15:54:24 [15:54:24.283]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f10ccc7cc5fb5cc6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f10ccc7cc5fb5cc6"}
15:54:24 [15:54:24.283]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0f4a7e9689edc8ed type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:0f4a7e9689edc8ed"}
15:54:24 [15:54:24.283]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19444a80054eac9d epochNumber=1
{"provingJobId":"1:BASE_PARITY:19444a80054eac9d"}
15:54:24 [15:54:24.284]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:19444a80054eac9d
{"provingJobId":"1:BASE_PARITY:19444a80054eac9d"}
15:54:24 [15:54:24.284]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:19444a80054eac9d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:19444a80054eac9d"}
15:54:24 [15:54:24.285]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7772458406479966 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7772458406479966"}
15:54:24 [15:54:24.285]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:19444a80054eac9d has already been completed
{"provingJobId":"1:BASE_PARITY:19444a80054eac9d"}
15:54:24 [15:54:24.286]
INFO:
prover-client:proving-broker Proving Broker started
15:54:24 [15:54:24.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:883ec5deb1b2d8e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:883ec5deb1b2d8e2"}
15:54:24 [15:54:24.287]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1491d55e55be42e9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1491d55e55be42e9"}
15:54:25 [15:54:25.338]
INFO:
prover-client:proving-broker Proving Broker started
15:54:25 [15:54:25.339]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:883ec5deb1b2d8e2 settled=false
{"provingJobId":"1:BASE_PARITY:883ec5deb1b2d8e2","status":"pending"}
15:54:25 [15:54:25.339]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1491d55e55be42e9 settled=false
{"provingJobId":"1:BASE_PARITY:1491d55e55be42e9","status":"pending"}
15:54:25 [15:54:25.340]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:883ec5deb1b2d8e2 type=BASE_PARITY startedAt=1749830064287 older agent has taken job
{"provingJobId":"1:BASE_PARITY:883ec5deb1b2d8e2"}
15:54:25 [15:54:25.340]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:883ec5deb1b2d8e2 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:883ec5deb1b2d8e2"}
15:54:25 [15:54:25.342]
INFO:
prover-client:proving-broker Proving Broker started
15:54:25 [15:54:25.343]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:257243a47bdeb782 epochNumber=1
{"provingJobId":"1:BASE_PARITY:257243a47bdeb782"}
15:54:25 [15:54:25.343]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3cb48ba6a8d40ece epochNumber=2
{"provingJobId":"1:BASE_PARITY:3cb48ba6a8d40ece"}
15:54:26 [15:54:26.344]
INFO:
prover-client:proving-broker Proving Broker started
15:54:26 [15:54:26.345]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:257243a47bdeb782 settled=false
{"provingJobId":"1:BASE_PARITY:257243a47bdeb782","status":"pending"}
15:54:26 [15:54:26.345]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3cb48ba6a8d40ece settled=false
{"provingJobId":"1:BASE_PARITY:3cb48ba6a8d40ece","status":"pending"}
15:54:26 [15:54:26.346]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:257243a47bdeb782 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:257243a47bdeb782"}
15:54:26 [15:54:26.347]
INFO:
prover-client:proving-broker Proving Broker started
15:54:26 [15:54:26.348]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d95a6f3ec2c6bf62 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d95a6f3ec2c6bf62"}
15:54:26 [15:54:26.348]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:322c212ff44a0e0d epochNumber=2
{"provingJobId":"1:BASE_PARITY:322c212ff44a0e0d"}
15:54:27 [15:54:27.349]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.350]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d95a6f3ec2c6bf62 settled=false
{"provingJobId":"1:BASE_PARITY:d95a6f3ec2c6bf62","status":"pending"}
15:54:27 [15:54:27.350]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:322c212ff44a0e0d settled=false
{"provingJobId":"1:BASE_PARITY:322c212ff44a0e0d","status":"pending"}
15:54:27 [15:54:27.351]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d95a6f3ec2c6bf62 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d95a6f3ec2c6bf62"}
15:54:27 [15:54:27.352]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d95a6f3ec2c6bf62 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d95a6f3ec2c6bf62"}
15:54:27 [15:54:27.353]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.354]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:649e622f08d86539 epochNumber=1
{"provingJobId":"1:BASE_PARITY:649e622f08d86539"}
15:54:27 [15:54:27.354]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7ab68e41a3f742e epochNumber=2
{"provingJobId":"1:BASE_PARITY:f7ab68e41a3f742e"}
15:54:27 [15:54:27.354]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:649e622f08d86539 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:649e622f08d86539"}
15:54:27 [15:54:27.355]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f7ab68e41a3f742e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f7ab68e41a3f742e"}
15:54:27 [15:54:27.356]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f71fcbb6b4b64f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f71fcbb6b4b64f8"}
15:54:27 [15:54:27.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4528211bb5bc0495 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4528211bb5bc0495"}
15:54:27 [15:54:27.357]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3f71fcbb6b4b64f8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3f71fcbb6b4b64f8"}
15:54:27 [15:54:27.357]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3f71fcbb6b4b64f8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3f71fcbb6b4b64f8"}
15:54:27 [15:54:27.358]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4528211bb5bc0495 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4528211bb5bc0495"}
15:54:27 [15:54:27.358]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.359]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6aebe67312f27f15 err=test error
{"provingJoId":"1:BASE_PARITY:6aebe67312f27f15"}
15:54:27 [15:54:27.359]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.360]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a62ddc5fad43d275 not found
{"provingJobId":"1:BASE_PARITY:a62ddc5fad43d275"}
15:54:27 [15:54:27.361]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.361]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1187dd643edf8c4f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1187dd643edf8c4f"}
15:54:27 [15:54:27.363]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.363]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af25a5b04150ac73 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af25a5b04150ac73"}
15:54:27 [15:54:27.464]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:af25a5b04150ac73 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:af25a5b04150ac73"}
15:54:27 [15:54:27.473]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.474]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a506a2f86e489260 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a506a2f86e489260"}
15:54:27 [15:54:27.575]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a506a2f86e489260 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a506a2f86e489260"}
15:54:27 [15:54:27.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e8a421fe43b9015 epochNumber=10
{"provingJobId":"1:BASE_PARITY:9e8a421fe43b9015"}
15:54:27 [15:54:27.601]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:27 [15:54:27.709]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.710]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed2f909bf91411f4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed2f909bf91411f4"}
15:54:27 [15:54:27.863]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ed2f909bf91411f4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:ed2f909bf91411f4"}
15:54:27 [15:54:27.912]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.913]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:226100fc6a12e1da epochNumber=1
{"provingJobId":"1:BASE_PARITY:226100fc6a12e1da"}
15:54:27 [15:54:27.914]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:226100fc6a12e1da type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:226100fc6a12e1da"}
15:54:27 [15:54:27.915]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.915]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17438747d4865603 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17438747d4865603"}
15:54:27 [15:54:27.916]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:17438747d4865603 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:17438747d4865603"}
15:54:27 [15:54:27.917]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:17438747d4865603 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:17438747d4865603"}
15:54:27 [15:54:27.917]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.918]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a5410db598dd0b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9a5410db598dd0b1"}
15:54:27 [15:54:27.918]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9a5410db598dd0b1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9a5410db598dd0b1"}
15:54:27 [15:54:27.919]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.919]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d61f9475507c84ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:d61f9475507c84ba"}
15:54:27 [15:54:27.920]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d61f9475507c84ba type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d61f9475507c84ba"}
15:54:27 [15:54:27.920]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:225ab469fe2e6527 epochNumber=3
{"provingJobId":"1:BASE_PARITY:225ab469fe2e6527"}
15:54:27 [15:54:27.944]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:27 [15:54:27.946]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d61f9475507c84ba err=test error
{"provingJoId":"1:BASE_PARITY:d61f9475507c84ba"}
15:54:27 [15:54:27.948]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.948]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3c07700fa2c5c9d0 settled=false
{"provingJobId":"1:BASE_PARITY:3c07700fa2c5c9d0","status":"pending"}
15:54:27 [15:54:27.948]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e5c3a0104356d7d7 settled=false
{"provingJobId":"1:BASE_PARITY:e5c3a0104356d7d7","status":"pending"}
15:54:27 [15:54:27.950]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.950]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4263b24964ec27a0 settled=true
{"provingJobId":"1:BASE_PARITY:4263b24964ec27a0","status":"fulfilled"}
15:54:27 [15:54:27.950]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:c1db084bfd421379 settled=true
{"provingJobId":"2:BASE_PARITY:c1db084bfd421379","status":"fulfilled"}
15:54:27 [15:54:27.951]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.951]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b4c96e989db75f01 settled=true
{"provingJobId":"1:BASE_PARITY:b4c96e989db75f01","status":"fulfilled"}
15:54:27 [15:54:27.951]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2cee9f0a5a11218e settled=false
{"provingJobId":"1:BASE_PARITY:2cee9f0a5a11218e","status":"pending"}
15:54:27 [15:54:27.952]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.953]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c214039bae4994ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:c214039bae4994ba"}
15:54:27 [15:54:27.955]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.955]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:560a1f14752638cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:560a1f14752638cd"}
15:54:27 [15:54:27.956]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:560a1f14752638cd: Error: db error: Error: db error
15:54:27 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:560a1f14752638cd"}
15:54:27 [15:54:27.957]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.957]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f021931c73d2aff0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f021931c73d2aff0"}
15:54:27 [15:54:27.958]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f021931c73d2aff0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f021931c73d2aff0"}
15:54:27 [15:54:27.958]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f021931c73d2aff0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f021931c73d2aff0"}
15:54:27 [15:54:27.959]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.960]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d363fde00bde0e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d363fde00bde0e2"}
15:54:27 [15:54:27.960]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4d363fde00bde0e2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4d363fde00bde0e2"}
15:54:27 [15:54:27.960]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4d363fde00bde0e2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4d363fde00bde0e2"}
15:54:27 [15:54:27.961]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:4d363fde00bde0e2: Error: db error
15:54:27 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:4d363fde00bde0e2"}
15:54:27 [15:54:27.962]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.962]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b0da29734b82ae0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b0da29734b82ae0"}
15:54:27 [15:54:27.963]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6b0da29734b82ae0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6b0da29734b82ae0"}
15:54:27 [15:54:27.963]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6b0da29734b82ae0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6b0da29734b82ae0"}
15:54:27 [15:54:27.964]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.965]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fbd1137745a6d60 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fbd1137745a6d60"}
15:54:27 [15:54:27.965]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0fbd1137745a6d60 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0fbd1137745a6d60"}
15:54:27 [15:54:27.965]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0fbd1137745a6d60 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0fbd1137745a6d60"}
15:54:27 [15:54:27.966]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:0fbd1137745a6d60 jobErr=test error: Error: db error
15:54:27 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:0fbd1137745a6d60"}
15:54:27 [15:54:27.967]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.968]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7dcd75bf56985df2 not found
{"provingJobId":"1:BASE_PARITY:7dcd75bf56985df2"}
15:54:27 [15:54:27.969]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.970]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d878b3c94d15f23d err=test error
{"provingJoId":"1:BASE_PARITY:d878b3c94d15f23d"}
15:54:27 [15:54:27.971]
INFO:
prover-client:proving-broker Proving Broker started
15:54:27 [15:54:27.981]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a4bc403149398ab epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a4bc403149398ab"}
15:54:27 [15:54:27.982]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1a4bc403149398ab type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1a4bc403149398ab"}
15:54:27 [15:54:27.982]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1a4bc403149398ab type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1a4bc403149398ab"}
15:54:27 [15:54:27.992]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:1b48fc3287e9787c epochNumber=2
{"provingJobId":"2:BASE_PARITY:1b48fc3287e9787c"}
15:54:27 [15:54:27.993]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:1b48fc3287e9787c type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:1b48fc3287e9787c"}
15:54:27 [15:54:27.993]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:1b48fc3287e9787c type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:1b48fc3287e9787c"}
15:54:28 [15:54:28.056]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:f44af07e24e9986b epochNumber=3
{"provingJobId":"3:BASE_PARITY:f44af07e24e9986b"}
15:54:28 [15:54:28.072]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:28 [15:54:28.124]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:cc519d73d76885f2 epochNumber=4
{"provingJobId":"4:BASE_PARITY:cc519d73d76885f2"}
15:54:28 [15:54:28.149]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:28 [15:54:28.187]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:2853cbc08f17ca48 epochNumber=5
{"provingJobId":"5:BASE_PARITY:2853cbc08f17ca48"}
15:54:28 [15:54:28.202]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:28 [15:54:28.241]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.242]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4cac9b7e54dca921 epochNumber=42
{"provingJobId":"1:BASE_PARITY:4cac9b7e54dca921"}
15:54:28 [15:54:28.245]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testvWb7vH/42 with map size 1073741824
15:54:28 [15:54:28.249]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.267]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:2bba256091e41163 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2bba256091e41163"}
15:54:28 [15:54:28.272]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90a7753a3869bb28 epochNumber=1
{"provingJobId":"1:BASE_PARITY:90a7753a3869bb28"}
15:54:28 [15:54:28.273]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt3bCBL/1 with map size 1073741824
15:54:28 [15:54:28.275]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.292]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:115406a62a4e69f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:115406a62a4e69f0"}
15:54:28 [15:54:28.300]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.301]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3fa4bbc2d14545ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:3fa4bbc2d14545ba"}
15:54:28 [15:54:28.301]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmKHO1c/1 with map size 1073741824
15:54:28 [15:54:28.305]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.324]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3fa4bbc2d14545ba epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3fa4bbc2d14545ba"}
15:54:28 [15:54:28.328]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.328]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04d4fa2d58bd87ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:04d4fa2d58bd87ba"}
15:54:28 [15:54:28.329]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoQNmwC/1 with map size 1073741824
15:54:28 [15:54:28.330]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.357]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:04d4fa2d58bd87ba epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:04d4fa2d58bd87ba"}
15:54:28 [15:54:28.358]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:04d4fa2d58bd87ba type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:04d4fa2d58bd87ba"}
15:54:28 [15:54:28.365]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:04d4fa2d58bd87ba epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:04d4fa2d58bd87ba"}
15:54:28 [15:54:28.368]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.369]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d7b23e351019b7f epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d7b23e351019b7f"}
15:54:28 [15:54:28.369]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwOhqDt/1 with map size 1073741824
15:54:28 [15:54:28.371]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.393]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.395]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.396]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a3e70458bd939eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a3e70458bd939eb"}
15:54:28 [15:54:28.396]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testk4UYdP/1 with map size 1073741824
15:54:28 [15:54:28.398]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.416]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7a3e70458bd939eb
{"provingJobId":"1:BASE_PARITY:7a3e70458bd939eb"}
15:54:28 [15:54:28.417]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7a3e70458bd939eb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7a3e70458bd939eb"}
15:54:28 [15:54:28.417]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7a3e70458bd939eb type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7a3e70458bd939eb"}
15:54:28 [15:54:28.426]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.427]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02c044801f442608 epochNumber=1
{"provingJobId":"1:BASE_PARITY:02c044801f442608"}
15:54:28 [15:54:28.428]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCHFgcE/1 with map size 1073741824
15:54:28 [15:54:28.430]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.448]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:02c044801f442608
{"provingJobId":"1:BASE_PARITY:02c044801f442608"}
15:54:28 [15:54:28.448]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:02c044801f442608 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:02c044801f442608"}
15:54:28 [15:54:28.467]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.468]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55f7dd4873f91bda epochNumber=1
{"provingJobId":"1:BASE_PARITY:55f7dd4873f91bda"}
15:54:28 [15:54:28.468]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testElEhDd/1 with map size 1073741824
15:54:28 [15:54:28.472]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.492]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:55f7dd4873f91bda type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:55f7dd4873f91bda"}
15:54:28 [15:54:28.493]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:55f7dd4873f91bda type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:55f7dd4873f91bda"}
15:54:28 [15:54:28.504]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.505]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e765b79fc98de73 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e765b79fc98de73"}
15:54:28 [15:54:28.505]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEwakuK/1 with map size 1073741824
15:54:28 [15:54:28.510]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.531]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7e765b79fc98de73 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7e765b79fc98de73"}
15:54:28 [15:54:28.531]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7e765b79fc98de73 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7e765b79fc98de73"}
15:54:28 [15:54:28.540]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.540]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:280294a263bd1ab1 epochNumber=0
{"provingJobId":"1:BASE_PARITY:280294a263bd1ab1"}
15:54:28 [15:54:28.541]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testePQ3F8/0 with map size 1073741824
15:54:28 [15:54:28.542]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.562]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:280294a263bd1ab1 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:280294a263bd1ab1"}
15:54:28 [15:54:28.563]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e20890f116cd161 epochNumber=0
{"provingJobId":"1:BASE_PARITY:8e20890f116cd161"}
15:54:28 [15:54:28.564]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:280294a263bd1ab1 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:280294a263bd1ab1"}
15:54:28 [15:54:28.564]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca556e6ac6abbb56 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ca556e6ac6abbb56"}
15:54:28 [15:54:28.579]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.582]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.583]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28bde7398a7568d0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:28bde7398a7568d0"}
15:54:28 [15:54:28.584]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test278REt/2 with map size 1073741824
15:54:28 [15:54:28.587]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76f01a0d4df083d8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:76f01a0d4df083d8"}
15:54:28 [15:54:28.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test278REt/1 with map size 1073741824
15:54:28 [15:54:28.610]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.630]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.630]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc4c639f5512066e epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc4c639f5512066e"}
15:54:28 [15:54:28.631]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcCuIaH/1 with map size 1073741824
15:54:28 [15:54:28.634]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.660]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.660]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28f4bc5b3476c8de epochNumber=1
{"provingJobId":"1:BASE_PARITY:28f4bc5b3476c8de"}
15:54:28 [15:54:28.661]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6ZdPtg/1 with map size 1073741824
15:54:28 [15:54:28.664]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.682]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19ce1652c03dc31f epochNumber=1
{"provingJobId":"1:BASE_PARITY:19ce1652c03dc31f"}
15:54:28 [15:54:28.689]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01fefe7d09cd475d epochNumber=2
{"provingJobId":"1:BASE_PARITY:01fefe7d09cd475d"}
15:54:28 [15:54:28.690]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6ZdPtg/2 with map size 1073741824
15:54:28 [15:54:28.691]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.709]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb2051354626765e epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb2051354626765e"}
15:54:28 [15:54:28.718]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.719]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f6c5cbebeb3a2ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f6c5cbebeb3a2ac"}
15:54:28 [15:54:28.719]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxsOQc6/1 with map size 1073741824
15:54:28 [15:54:28.722]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.739]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ec5683b22b9c4cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ec5683b22b9c4cd"}
15:54:28 [15:54:28.748]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b04235528295818 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9b04235528295818"}
15:54:28 [15:54:28.749]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testxsOQc6/2 with map size 1073741824
15:54:28 [15:54:28.752]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14196c361e9d1697 epochNumber=1
{"provingJobId":"1:BASE_PARITY:14196c361e9d1697"}
15:54:28 [15:54:28.781]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98d648fb3ce49669 epochNumber=1
{"provingJobId":"1:BASE_PARITY:98d648fb3ce49669"}
15:54:28 [15:54:28.782]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCSCtvr/1 with map size 1073741824
15:54:28 [15:54:28.784]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4d83b690b6c5c88 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4d83b690b6c5c88"}
15:54:28 [15:54:28.810]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:073bc2493345fb32 epochNumber=2
{"provingJobId":"1:BASE_PARITY:073bc2493345fb32"}
15:54:28 [15:54:28.811]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCSCtvr/2 with map size 1073741824
15:54:28 [15:54:28.814]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.833]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e91d84a345974c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e91d84a345974c1"}
15:54:28 [15:54:28.844]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.845]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f0f5e4c2492b614 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f0f5e4c2492b614"}
15:54:28 [15:54:28.846]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testg40jXr/1 with map size 1073741824
15:54:28 [15:54:28.849]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.869]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed35a0f92604c921 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed35a0f92604c921"}
15:54:28 [15:54:28.875]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2f0f5e4c2492b614 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2f0f5e4c2492b614"}
15:54:28 [15:54:28.885]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.886]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55bcacb545eb9293 epochNumber=1
{"provingJobId":"1:BASE_PARITY:55bcacb545eb9293"}
15:54:28 [15:54:28.887]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH7jRvb/1 with map size 1073741824
15:54:28 [15:54:28.889]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.906]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:138242aeff6b4c8f epochNumber=1
{"provingJobId":"1:BASE_PARITY:138242aeff6b4c8f"}
15:54:28 [15:54:28.912]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:55bcacb545eb9293 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:55bcacb545eb9293"}
15:54:28 [15:54:28.922]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.923]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fd1348c354cd6c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fd1348c354cd6c9"}
15:54:28 [15:54:28.923]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmwIE0D/1 with map size 1073741824
15:54:28 [15:54:28.925]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f839e061e3545201 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f839e061e3545201"}
15:54:28 [15:54:28.955]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0fd1348c354cd6c9 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:0fd1348c354cd6c9"}
15:54:28 [15:54:28.961]
INFO:
prover-client:proving-broker Proving Broker started
15:54:28 [15:54:28.962]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dac9a79e2a4211d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dac9a79e2a4211d7"}
15:54:28 [15:54:28.963]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt8lrnO/1 with map size 1073741824
15:54:28 [15:54:28.965]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:28 [15:54:28.984]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:dac9a79e2a4211d7
{"provingJobId":"1:BASE_PARITY:dac9a79e2a4211d7"}
15:54:28 [15:54:28.985]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dac9a79e2a4211d7 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:dac9a79e2a4211d7"}
15:54:28 [15:54:28.991]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a7e1abd04165a09d epochNumber=1
{"provingJobId":"1:BASE_PARITY:a7e1abd04165a09d"}
15:54:28 [15:54:28.996]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dac9a79e2a4211d7 has already been completed
{"provingJobId":"1:BASE_PARITY:dac9a79e2a4211d7"}
15:54:29 [15:54:29.002]
INFO:
prover-client:proving-broker Proving Broker started
15:54:29 [15:54:29.003]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:706f565b7311d5a9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:706f565b7311d5a9"}
15:54:29 [15:54:29.004]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrBFX4S/1 with map size 1073741824
15:54:29 [15:54:29.005]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:29 [15:54:29.022]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9978b0560ccf7cda epochNumber=2
{"provingJobId":"1:BASE_PARITY:9978b0560ccf7cda"}
15:54:29 [15:54:29.023]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrBFX4S/2 with map size 1073741824
15:54:29 [15:54:29.027]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:30 [15:54:30.100]
INFO:
prover-client:proving-broker Proving Broker started
15:54:30 [15:54:30.106]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:706f565b7311d5a9 settled=false
{"provingJobId":"1:BASE_PARITY:706f565b7311d5a9","status":"pending"}
15:54:30 [15:54:30.107]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9978b0560ccf7cda settled=false
{"provingJobId":"1:BASE_PARITY:9978b0560ccf7cda","status":"pending"}
15:54:30 [15:54:30.108]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:706f565b7311d5a9 type=BASE_PARITY startedAt=1749830069048 older agent has taken job
{"provingJobId":"1:BASE_PARITY:706f565b7311d5a9"}
15:54:30 [15:54:30.108]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:706f565b7311d5a9 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:706f565b7311d5a9"}
15:54:30 [15:54:30.114]
INFO:
prover-client:proving-broker Proving Broker started
15:54:30 [15:54:30.114]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d540b24483954ed8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d540b24483954ed8"}
15:54:30 [15:54:30.115]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcHyj3E/1 with map size 1073741824
15:54:30 [15:54:30.116]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:30 [15:54:30.134]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85bce1b4e1d74b6d epochNumber=2
{"provingJobId":"1:BASE_PARITY:85bce1b4e1d74b6d"}
15:54:30 [15:54:30.135]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcHyj3E/2 with map size 1073741824
15:54:30 [15:54:30.144]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:31 [15:54:31.168]
INFO:
prover-client:proving-broker Proving Broker started
15:54:31 [15:54:31.171]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d540b24483954ed8 settled=false
{"provingJobId":"1:BASE_PARITY:d540b24483954ed8","status":"pending"}
15:54:31 [15:54:31.172]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:85bce1b4e1d74b6d settled=false
{"provingJobId":"1:BASE_PARITY:85bce1b4e1d74b6d","status":"pending"}
15:54:31 [15:54:31.173]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d540b24483954ed8 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:d540b24483954ed8"}
15:54:31 [15:54:31.176]
INFO:
prover-client:proving-broker Proving Broker started
15:54:31 [15:54:31.177]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36b98c7e2bbb5a11 epochNumber=1
{"provingJobId":"1:BASE_PARITY:36b98c7e2bbb5a11"}
15:54:31 [15:54:31.178]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8tC8Lv/1 with map size 1073741824
15:54:31 [15:54:31.184]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:31 [15:54:31.202]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab3d0489914809d4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ab3d0489914809d4"}
15:54:31 [15:54:31.203]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8tC8Lv/2 with map size 1073741824
15:54:31 [15:54:31.209]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:32 [15:54:32.240]
INFO:
prover-client:proving-broker Proving Broker started
15:54:32 [15:54:32.255]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:36b98c7e2bbb5a11 settled=false
{"provingJobId":"1:BASE_PARITY:36b98c7e2bbb5a11","status":"pending"}
15:54:32 [15:54:32.262]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ab3d0489914809d4 settled=false
{"provingJobId":"1:BASE_PARITY:ab3d0489914809d4","status":"pending"}
15:54:32 [15:54:32.265]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:36b98c7e2bbb5a11 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:36b98c7e2bbb5a11"}
15:54:32 [15:54:32.266]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:36b98c7e2bbb5a11 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:36b98c7e2bbb5a11"}
15:54:32 [15:54:32.280]
INFO:
prover-client:proving-broker Proving Broker started
15:54:32 [15:54:32.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3cbf1ca431f74227 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3cbf1ca431f74227"}
15:54:32 [15:54:32.282]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEegYZh/1 with map size 1073741824
15:54:32 [15:54:32.284]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:32 [15:54:32.305]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01df43ff10d9fbba epochNumber=2
{"provingJobId":"1:BASE_PARITY:01df43ff10d9fbba"}
15:54:32 [15:54:32.309]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testEegYZh/2 with map size 1073741824
15:54:32 [15:54:32.311]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:32 [15:54:32.334]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3cbf1ca431f74227 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3cbf1ca431f74227"}
15:54:32 [15:54:32.345]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:01df43ff10d9fbba type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:01df43ff10d9fbba"}
15:54:32 [15:54:32.365]
INFO:
prover-client:proving-broker Proving Broker started
15:54:32 [15:54:32.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:894f65ebe9e2742b epochNumber=1
{"provingJobId":"1:BASE_PARITY:894f65ebe9e2742b"}
15:54:32 [15:54:32.368]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testX7C1wc/1 with map size 1073741824
15:54:32 [15:54:32.370]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:32 [15:54:32.394]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71efdc3257e3f97e epochNumber=2
{"provingJobId":"1:BASE_PARITY:71efdc3257e3f97e"}
15:54:32 [15:54:32.396]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testX7C1wc/2 with map size 1073741824
15:54:32 [15:54:32.401]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:32 [15:54:32.420]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:894f65ebe9e2742b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:894f65ebe9e2742b"}
15:54:32 [15:54:32.422]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:894f65ebe9e2742b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:894f65ebe9e2742b"}
15:54:32 [15:54:32.429]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:71efdc3257e3f97e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:71efdc3257e3f97e"}
15:54:32 [15:54:32.448]
INFO:
prover-client:proving-broker Proving Broker started
15:54:32 [15:54:32.451]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7f8ed2dfe2ac396e err=test error
{"provingJoId":"1:BASE_PARITY:7f8ed2dfe2ac396e"}
15:54:32 [15:54:32.460]
INFO:
prover-client:proving-broker Proving Broker started
15:54:32 [15:54:32.462]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9b8ed3d519370aa0 not found
{"provingJobId":"1:BASE_PARITY:9b8ed3d519370aa0"}
15:54:32 [15:54:32.468]
INFO:
prover-client:proving-broker Proving Broker started
15:54:32 [15:54:32.469]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08104abe5ea4caad epochNumber=1
{"provingJobId":"1:BASE_PARITY:08104abe5ea4caad"}
15:54:32 [15:54:32.473]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwJ02jk/1 with map size 1073741824
15:54:32 [15:54:32.475]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:32 [15:54:32.504]
INFO:
prover-client:proving-broker Proving Broker started
15:54:32 [15:54:32.505]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9553d91349b82093 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9553d91349b82093"}
15:54:32 [15:54:32.507]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKe4Jft/1 with map size 1073741824
15:54:32 [15:54:32.509]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:32 [15:54:32.633]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9553d91349b82093 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9553d91349b82093"}
15:54:32 [15:54:32.646]
INFO:
prover-client:proving-broker Proving Broker started
15:54:32 [15:54:32.649]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28b50dd5498e7d5b epochNumber=1
{"provingJobId":"1:BASE_PARITY:28b50dd5498e7d5b"}
15:54:32 [15:54:32.652]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFwDhQE/1 with map size 1073741824
15:54:32 [15:54:32.658]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:32 [15:54:32.800]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:28b50dd5498e7d5b timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:28b50dd5498e7d5b"}
15:54:32 [15:54:32.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19d15d8cc8afccfe epochNumber=10
{"provingJobId":"1:BASE_PARITY:19d15d8cc8afccfe"}
15:54:32 [15:54:32.807]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testFwDhQE/10 with map size 1073741824
15:54:32 [15:54:32.811]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:32 [15:54:32.854]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:32 [15:54:32.855]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:54:32 [15:54:32.965]
INFO:
prover-client:proving-broker Proving Broker started
15:54:32 [15:54:32.966]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5789730610375f80 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5789730610375f80"}
15:54:32 [15:54:32.967]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrzI4df/1 with map size 1073741824
15:54:32 [15:54:32.969]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.145]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5789730610375f80 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5789730610375f80"}
15:54:33 [15:54:33.200]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.202]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0d4814850ad66e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0d4814850ad66e2"}
15:54:33 [15:54:33.203]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5pgB1J/1 with map size 1073741824
15:54:33 [15:54:33.207]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.232]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b0d4814850ad66e2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b0d4814850ad66e2"}
15:54:33 [15:54:33.239]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.244]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bcd9a19043b5d06e epochNumber=1
{"provingJobId":"1:BASE_PARITY:bcd9a19043b5d06e"}
15:54:33 [15:54:33.248]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqW0ko0/1 with map size 1073741824
15:54:33 [15:54:33.258]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.293]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bcd9a19043b5d06e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:bcd9a19043b5d06e"}
15:54:33 [15:54:33.296]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bcd9a19043b5d06e type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:bcd9a19043b5d06e"}
15:54:33 [15:54:33.306]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.307]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb90c47b7b0597e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb90c47b7b0597e3"}
15:54:33 [15:54:33.311]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFdTgsp/1 with map size 1073741824
15:54:33 [15:54:33.316]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.348]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fb90c47b7b0597e3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fb90c47b7b0597e3"}
15:54:33 [15:54:33.368]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.370]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13315d1f865a80c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13315d1f865a80c4"}
15:54:33 [15:54:33.371]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWOx26T/1 with map size 1073741824
15:54:33 [15:54:33.383]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.409]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:13315d1f865a80c4 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:13315d1f865a80c4"}
15:54:33 [15:54:33.410]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d2c62b42420a0f9 epochNumber=3
{"provingJobId":"1:BASE_PARITY:6d2c62b42420a0f9"}
15:54:33 [15:54:33.411]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testWOx26T/3 with map size 1073741824
15:54:33 [15:54:33.415]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.445]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:33 [15:54:33.446]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:54:33 [15:54:33.465]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:13315d1f865a80c4 err=test error
{"provingJoId":"1:BASE_PARITY:13315d1f865a80c4"}
15:54:33 [15:54:33.474]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQl0m4O/1 with map size 1073741824
15:54:33 [15:54:33.478]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.502]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQl0m4O/2 with map size 1073741824
15:54:33 [15:54:33.506]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.534]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.542]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8f0f7f9378bbae7d settled=false
{"provingJobId":"1:BASE_PARITY:8f0f7f9378bbae7d","status":"pending"}
15:54:33 [15:54:33.546]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3aa91e4fbc039d3f settled=false
{"provingJobId":"1:BASE_PARITY:3aa91e4fbc039d3f","status":"pending"}
15:54:33 [15:54:33.572]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testh1iWea/1 with map size 1073741824
15:54:33 [15:54:33.576]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.600]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testh1iWea/2 with map size 1073741824
15:54:33 [15:54:33.617]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.681]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.687]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9a1d5cec41abd7f5 settled=true
{"provingJobId":"1:BASE_PARITY:9a1d5cec41abd7f5","status":"fulfilled"}
15:54:33 [15:54:33.691]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:40461b5612582580 settled=true
{"provingJobId":"2:BASE_PARITY:40461b5612582580","status":"fulfilled"}
15:54:33 [15:54:33.704]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFoiqw1/1 with map size 1073741824
15:54:33 [15:54:33.709]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.762]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFoiqw1/2 with map size 1073741824
15:54:33 [15:54:33.768]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.791]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.793]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d7ab296770e29249 settled=true
{"provingJobId":"1:BASE_PARITY:d7ab296770e29249","status":"fulfilled"}
15:54:33 [15:54:33.796]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:69863112edd6c09f settled=false
{"provingJobId":"1:BASE_PARITY:69863112edd6c09f","status":"pending"}
15:54:33 [15:54:33.806]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f0acac5bc06d7e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f0acac5bc06d7e1"}
15:54:33 [15:54:33.808]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrPhTg8/1 with map size 1073741824
15:54:33 [15:54:33.811]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.844]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e18f83fab5110b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e18f83fab5110b0"}
15:54:33 [15:54:33.847]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:3e18f83fab5110b0: Error: db error: Error: db error
15:54:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:3e18f83fab5110b0"}
15:54:33 [15:54:33.861]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9e8ece4bb8b351e epochNumber=1
{"provingJobId":"1:BASE_PARITY:a9e8ece4bb8b351e"}
15:54:33 [15:54:33.863]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testn983Qp/1 with map size 1073741824
15:54:33 [15:54:33.867]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.885]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a9e8ece4bb8b351e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a9e8ece4bb8b351e"}
15:54:33 [15:54:33.885]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a9e8ece4bb8b351e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a9e8ece4bb8b351e"}
15:54:33 [15:54:33.898]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.898]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7fab1ab9aa292522 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7fab1ab9aa292522"}
15:54:33 [15:54:33.899]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLOSZHn/1 with map size 1073741824
15:54:33 [15:54:33.903]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.924]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7fab1ab9aa292522 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7fab1ab9aa292522"}
15:54:33 [15:54:33.925]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7fab1ab9aa292522 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7fab1ab9aa292522"}
15:54:33 [15:54:33.926]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:7fab1ab9aa292522: Error: db error
15:54:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:7fab1ab9aa292522"}
15:54:33 [15:54:33.934]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7d585262d44a8ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:d7d585262d44a8ff"}
15:54:33 [15:54:33.936]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfi47uE/1 with map size 1073741824
15:54:33 [15:54:33.947]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:33 [15:54:33.975]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d7d585262d44a8ff type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d7d585262d44a8ff"}
15:54:33 [15:54:33.976]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d7d585262d44a8ff type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d7d585262d44a8ff"}
15:54:33 [15:54:33.991]
INFO:
prover-client:proving-broker Proving Broker started
15:54:33 [15:54:33.992]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:937662c7c15b8298 epochNumber=1
{"provingJobId":"1:BASE_PARITY:937662c7c15b8298"}
15:54:33 [15:54:33.993]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH5EP2A/1 with map size 1073741824
15:54:33 [15:54:33.999]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:34 [15:54:34.021]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:937662c7c15b8298 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:937662c7c15b8298"}
15:54:34 [15:54:34.022]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:937662c7c15b8298 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:937662c7c15b8298"}
15:54:34 [15:54:34.022]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:937662c7c15b8298 jobErr=test error: Error: db error
15:54:34 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:937662c7c15b8298"}
15:54:34 [15:54:34.027]
INFO:
prover-client:proving-broker Proving Broker started
15:54:34 [15:54:34.027]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aacd109c4100188e not found
{"provingJobId":"1:BASE_PARITY:aacd109c4100188e"}
15:54:34 [15:54:34.033]
INFO:
prover-client:proving-broker Proving Broker started
15:54:34 [15:54:34.033]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b87740b9f6cc145a err=test error
{"provingJoId":"1:BASE_PARITY:b87740b9f6cc145a"}
15:54:34 [15:54:34.041]
INFO:
prover-client:proving-broker Proving Broker started
15:54:34 [15:54:34.052]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90a6b720035a0805 epochNumber=1
{"provingJobId":"1:BASE_PARITY:90a6b720035a0805"}
15:54:34 [15:54:34.057]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBVQxwZ/1 with map size 1073741824
15:54:34 [15:54:34.062]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:34 [15:54:34.079]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:90a6b720035a0805 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:90a6b720035a0805"}
15:54:34 [15:54:34.080]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:90a6b720035a0805 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:90a6b720035a0805"}
15:54:34 [15:54:34.098]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:c1ae51fe2cb0bf51 epochNumber=2
{"provingJobId":"2:BASE_PARITY:c1ae51fe2cb0bf51"}
15:54:34 [15:54:34.101]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testBVQxwZ/2 with map size 1073741824
15:54:34 [15:54:34.105]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:34 [15:54:34.129]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:c1ae51fe2cb0bf51 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:c1ae51fe2cb0bf51"}
15:54:34 [15:54:34.129]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c1ae51fe2cb0bf51 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:c1ae51fe2cb0bf51"}
15:54:34 [15:54:34.199]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:d83b8339bb950837 epochNumber=3
{"provingJobId":"3:BASE_PARITY:d83b8339bb950837"}
15:54:34 [15:54:34.203]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testBVQxwZ/3 with map size 1073741824
15:54:34 [15:54:34.210]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:34 [15:54:34.253]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:34 [15:54:34.254]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:54:34 [15:54:34.296]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:36fbb077463d9d03 epochNumber=4
{"provingJobId":"4:BASE_PARITY:36fbb077463d9d03"}
15:54:34 [15:54:34.301]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testBVQxwZ/4 with map size 1073741824
15:54:34 [15:54:34.305]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:34 [15:54:34.335]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:34 [15:54:34.336]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
15:54:34 [15:54:34.389]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:1b6c3e9aa8550b26 epochNumber=5
{"provingJobId":"5:BASE_PARITY:1b6c3e9aa8550b26"}
15:54:34 [15:54:34.390]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testBVQxwZ/5 with map size 1073741824
15:54:34 [15:54:34.392]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:54:34 [15:54:34.423]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:54:34 [15:54:34.423]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
15:54:34
PASS src/proving_broker/proving_broker.test.ts (
13.055 s)
15:54:34 ProvingBroker
15:54:34 Producer API
15:54:34
✓ refuses stale jobs (38 ms)
15:54:34
✓ enqueues jobs (4 ms)
15:54:34
✓ ignores duplicate jobs (4 ms)
15:54:34
✓ reports correct status when enqueuing repeat jobs (4 ms)
15:54:34
✓ throws an error in case of duplicate job IDs (22 ms)
15:54:34
✓ returns not-found status for non-existing jobs (1 ms)
15:54:34
✓ cancels jobs in queue (3 ms)
15:54:34
✓ cancels jobs in-progress (2 ms)
15:54:34
✓ returns job result if successful (9 ms)
15:54:34
✓ returns job error if failed (2 ms)
15:54:34
✓ correctly returns job status for concurrent writes (2 ms)
15:54:34
✓ refuses stale jobs (31 ms)
15:54:34
✓ enqueues jobs (29 ms)
15:54:34
✓ ignores duplicate jobs (27 ms)
15:54:34
✓ reports correct status when enqueuing repeat jobs (40 ms)
15:54:34
✓ throws an error in case of duplicate job IDs (24 ms)
15:54:34
✓ returns not-found status for non-existing jobs (2 ms)
15:54:34
✓ cancels jobs in queue (31 ms)
15:54:34
✓ cancels jobs in-progress (38 ms)
15:54:34
✓ returns job result if successful (39 ms)
15:54:34
✓ returns job error if failed (35 ms)
15:54:34
✓ correctly returns job status for concurrent writes (38 ms)
15:54:34 Consumer API
15:54:34
✓ returns undefined if no jobs are available (1 ms)
15:54:34
✓ returns jobs in priority order (1 ms)
15:54:34
✓ returns undefined if no jobs are available for the given allowList (1 ms)
15:54:34
✓ returns a job if it is in the allowList (3 ms)
15:54:34
✓ returns the most important job if it is in the allowList (1 ms)
15:54:34
✓ returns any job if filter is empty (2 ms)
15:54:34
✓ returns a new job when reporting job success (2 ms)
15:54:34
✓ returns a new job when reporting permanent error (2 ms)
15:54:34
✓ returns a new job when reporting retry-able error (1 ms)
15:54:34
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
15:54:34
✓ returns a new job if job is already in progress elsewhere (1056 ms)
15:54:34
✓ avoids sending the same job to a new agent after a restart (1005 ms)
15:54:34
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
15:54:34
✓ tracks job result if in progress (2 ms)
15:54:34
✓ tracks job result even if job is in queue (2 ms)
15:54:34
✓ ignores reported job error if unknown job (1 ms)
15:54:34
✓ ignores job result if unknown job (1 ms)
15:54:34
✓ returns undefined if no jobs are available (2 ms)
15:54:34
✓ returns jobs in priority order (47 ms)
15:54:34
✓ returns undefined if no jobs are available for the given allowList (30 ms)
15:54:34
✓ returns a job if it is in the allowList (58 ms)
15:54:34
✓ returns the most important job if it is in the allowList (62 ms)
15:54:34
✓ returns any job if filter is empty (63 ms)
15:54:34
✓ returns a new job when reporting job success (40 ms)
15:54:34
✓ returns a new job when reporting permanent error (37 ms)
15:54:34
✓ returns a new job when reporting retry-able error (37 ms)
15:54:34
✓ returns a new job when reporting progress if current one is cancelled (41 ms)
15:54:34
✓ returns a new job if job is already in progress elsewhere (1110 ms)
15:54:34
✓ avoids sending the same job to a new agent after a restart (1064 ms)
15:54:34
✓ avoids sending a completed job to a new agent after a restart (1101 ms)
15:54:34
✓ tracks job result if in progress (83 ms)
15:54:34
✓ tracks job result even if job is in queue (82 ms)
15:54:34
✓ ignores reported job error if unknown job (13 ms)
15:54:34
✓ ignores job result if unknown job (5 ms)
15:54:34 Timeouts
15:54:34
✓ tracks in progress jobs (1 ms)
15:54:34
✓ re-enqueues jobs that time out (110 ms)
15:54:34
✓ cancel stale jobs that time out (237 ms)
15:54:34
✓ keeps the jobs in progress while it is alive (202 ms)
15:54:34
✓ tracks in progress jobs (32 ms)
15:54:34
✓ re-enqueues jobs that time out (141 ms)
15:54:34
✓ cancel stale jobs that time out (321 ms)
15:54:34
✓ keeps the jobs in progress while it is alive (234 ms)
15:54:34 Retries
15:54:34
✓ retries jobs (3 ms)
15:54:34
✓ retries up to a maximum number of times (2 ms)
15:54:34
✓ passing retry=false does not retry (2 ms)
15:54:34
✓ does not retry if job is stale (28 ms)
15:54:34
✓ retries jobs (38 ms)
15:54:34
✓ retries up to a maximum number of times (68 ms)
15:54:34
✓ passing retry=false does not retry (62 ms)
15:54:34
✓ does not retry if job is stale (103 ms)
15:54:34 Database management
15:54:34
✓ re-enqueues proof requests on start (2 ms)
15:54:34
✓ restores proof results on start (1 ms)
15:54:34
✓ only re-enqueues unfinished jobs (1 ms)
15:54:34
✓ saves job when enqueued (3 ms)
15:54:34
✓ does not retain job if database fails to save (2 ms)
15:54:34
✓ saves job result (2 ms)
15:54:34
✓ saves result even if database fails to save (2 ms)
15:54:34
✓ saves job error (2 ms)
15:54:34
✓ saves job error even if database fails to save (3 ms)
15:54:34
✓ does not save job result if job is unknown (1 ms)
15:54:34
✓ does not save job error if job is unknown (1 ms)
15:54:34
✓ cleans up old jobs periodically (269 ms)
15:54:34
✓ re-enqueues proof requests on start (93 ms)
15:54:34
✓ restores proof results on start (133 ms)
15:54:34
✓ only re-enqueues unfinished jobs (104 ms)
15:54:34
✓ saves job when enqueued (31 ms)
15:54:34
✓ does not retain job if database fails to save (15 ms)
15:54:34
✓ saves job result (42 ms)
15:54:34
✓ saves result even if database fails to save (34 ms)
15:54:34
✓ saves job error (57 ms)
15:54:34
✓ saves job error even if database fails to save (37 ms)
15:54:34
✓ does not save job result if job is unknown (3 ms)
15:54:34
✓ does not save job error if job is unknown (5 ms)
15:54:34
✓ cleans up old jobs periodically (434 ms)
15:54:34
15:54:34
Test Suites: 1 passed, 1 total
15:54:34
Tests: 96 passed, 96 total
15:54:34
Snapshots: 0 total
15:54:34
Time: 13.158 s
15:54:34
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
15:54:34
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?