Parent Log:
http://ci.aztec-labs.com/990a96605b59e71e
Command: f409422f407df62d yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/f5bf74cc7cfdd8f2c3b0a69b410ed0dd2e068cda
Env: REF_NAME=gh-readonly-queue/next/pr-14610-d5c48d855d83e19712c6b72281801c6fce176ec1 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Wed Jun 18 14:01:19 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14610_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
14:01:24 [14:01:24.155]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.163]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:524187a11d944679 epochNumber=42
{"provingJobId":"1:BASE_PARITY:524187a11d944679"}
14:01:24 [14:01:24.166]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:d25edc5a00681e07 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d25edc5a00681e07"}
14:01:24 [14:01:24.194]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.196]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b4806a7799d9cbb8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b4806a7799d9cbb8"}
14:01:24 [14:01:24.197]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1b7c0c457f205b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1b7c0c457f205b1"}
14:01:24 [14:01:24.200]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.201]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4e1fcec4e9684ae epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4e1fcec4e9684ae"}
14:01:24 [14:01:24.203]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c4e1fcec4e9684ae epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c4e1fcec4e9684ae"}
14:01:24 [14:01:24.205]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.206]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d3e190cce8a2a01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d3e190cce8a2a01"}
14:01:24 [14:01:24.208]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9d3e190cce8a2a01 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9d3e190cce8a2a01"}
14:01:24 [14:01:24.208]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9d3e190cce8a2a01 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9d3e190cce8a2a01"}
14:01:24 [14:01:24.209]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9d3e190cce8a2a01 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9d3e190cce8a2a01"}
14:01:24 [14:01:24.211]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.211]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19435df74db38546 epochNumber=1
{"provingJobId":"1:BASE_PARITY:19435df74db38546"}
14:01:24 [14:01:24.241]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.243]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.243]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93db12fcb407f8b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:93db12fcb407f8b3"}
14:01:24 [14:01:24.244]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:93db12fcb407f8b3
{"provingJobId":"1:BASE_PARITY:93db12fcb407f8b3"}
14:01:24 [14:01:24.245]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:93db12fcb407f8b3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:93db12fcb407f8b3"}
14:01:24 [14:01:24.245]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:93db12fcb407f8b3 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:93db12fcb407f8b3"}
14:01:24 [14:01:24.247]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.247]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53731a3f70d6b145 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53731a3f70d6b145"}
14:01:24 [14:01:24.248]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:53731a3f70d6b145
{"provingJobId":"1:BASE_PARITY:53731a3f70d6b145"}
14:01:24 [14:01:24.248]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:53731a3f70d6b145 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:53731a3f70d6b145"}
14:01:24 [14:01:24.249]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:228ba4b7561577c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:228ba4b7561577c3"}
14:01:24 [14:01:24.251]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:228ba4b7561577c3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:228ba4b7561577c3"}
14:01:24 [14:01:24.251]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:228ba4b7561577c3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:228ba4b7561577c3"}
14:01:24 [14:01:24.252]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9df089f5c68bcc94 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9df089f5c68bcc94"}
14:01:24 [14:01:24.253]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9df089f5c68bcc94 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9df089f5c68bcc94"}
14:01:24 [14:01:24.253]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9df089f5c68bcc94 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9df089f5c68bcc94"}
14:01:24 [14:01:24.255]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4831596f3dd1e14a epochNumber=0
{"provingJobId":"1:BASE_PARITY:4831596f3dd1e14a"}
14:01:24 [14:01:24.255]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4831596f3dd1e14a epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4831596f3dd1e14a"}
14:01:24 [14:01:24.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8dc524aef0bfa29e epochNumber=0
{"provingJobId":"1:BASE_PARITY:8dc524aef0bfa29e"}
14:01:24 [14:01:24.256]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4831596f3dd1e14a epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4831596f3dd1e14a"}
14:01:24 [14:01:24.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c8e0ed487f7ed42 epochNumber=0
{"provingJobId":"1:BASE_PARITY:5c8e0ed487f7ed42"}
14:01:24 [14:01:24.258]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.276]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aab3fad0c6a02753 epochNumber=2
{"provingJobId":"1:BASE_PARITY:aab3fad0c6a02753"}
14:01:24 [14:01:24.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:383f093006045bb8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:383f093006045bb8"}
14:01:24 [14:01:24.279]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8e1e6968e5a2721 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e8e1e6968e5a2721"}
14:01:24 [14:01:24.281]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1895778d217f6380 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1895778d217f6380"}
14:01:24 [14:01:24.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dde3d9b2710606dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:dde3d9b2710606dd"}
14:01:24 [14:01:24.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:336f3684be5654fb epochNumber=2
{"provingJobId":"1:BASE_PARITY:336f3684be5654fb"}
14:01:24 [14:01:24.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e69b651563bd613 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e69b651563bd613"}
14:01:24 [14:01:24.285]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d84be29f6ab9d368 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d84be29f6ab9d368"}
14:01:24 [14:01:24.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3180b7af9284d19 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3180b7af9284d19"}
14:01:24 [14:01:24.287]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ced3ca9fece960ae epochNumber=2
{"provingJobId":"1:BASE_PARITY:ced3ca9fece960ae"}
14:01:24 [14:01:24.288]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b6698d1a7800afb epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b6698d1a7800afb"}
14:01:24 [14:01:24.290]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.290]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f33c839df2ebb3ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:f33c839df2ebb3ec"}
14:01:24 [14:01:24.291]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f802ab987470748a epochNumber=1
{"provingJobId":"1:BASE_PARITY:f802ab987470748a"}
14:01:24 [14:01:24.291]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f26da759352664b epochNumber=2
{"provingJobId":"1:BASE_PARITY:0f26da759352664b"}
14:01:24 [14:01:24.292]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:495fd0dcd59c0c10 epochNumber=1
{"provingJobId":"1:BASE_PARITY:495fd0dcd59c0c10"}
14:01:24 [14:01:24.293]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.294]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f449f6405fe5acb epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f449f6405fe5acb"}
14:01:24 [14:01:24.295]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf03e07460641f02 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf03e07460641f02"}
14:01:24 [14:01:24.295]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5f449f6405fe5acb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5f449f6405fe5acb"}
14:01:24 [14:01:24.297]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.297]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53876fc599152464 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53876fc599152464"}
14:01:24 [14:01:24.298]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be00db653b89eea4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:be00db653b89eea4"}
14:01:24 [14:01:24.298]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:53876fc599152464 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:53876fc599152464"}
14:01:24 [14:01:24.299]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.300]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a808211b68fa03e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a808211b68fa03e6"}
14:01:24 [14:01:24.300]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a39026c644b11fb2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a39026c644b11fb2"}
14:01:24 [14:01:24.301]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a808211b68fa03e6 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:a808211b68fa03e6"}
14:01:24 [14:01:24.302]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.303]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d484118e7ca3b8de epochNumber=1
{"provingJobId":"1:BASE_PARITY:d484118e7ca3b8de"}
14:01:24 [14:01:24.303]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d484118e7ca3b8de
{"provingJobId":"1:BASE_PARITY:d484118e7ca3b8de"}
14:01:24 [14:01:24.304]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d484118e7ca3b8de type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:d484118e7ca3b8de"}
14:01:24 [14:01:24.304]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06373cdd7e5dab3e epochNumber=1
{"provingJobId":"1:BASE_PARITY:06373cdd7e5dab3e"}
14:01:24 [14:01:24.305]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d484118e7ca3b8de has already been completed
{"provingJobId":"1:BASE_PARITY:d484118e7ca3b8de"}
14:01:24 [14:01:24.306]
INFO:
prover-client:proving-broker Proving Broker started
14:01:24 [14:01:24.307]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d22aa22a8112500 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d22aa22a8112500"}
14:01:24 [14:01:24.307]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:525f564541551653 epochNumber=2
{"provingJobId":"1:BASE_PARITY:525f564541551653"}
14:01:25 [14:01:25.359]
INFO:
prover-client:proving-broker Proving Broker started
14:01:25 [14:01:25.360]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9d22aa22a8112500 settled=false
{"provingJobId":"1:BASE_PARITY:9d22aa22a8112500","status":"pending"}
14:01:25 [14:01:25.360]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:525f564541551653 settled=false
{"provingJobId":"1:BASE_PARITY:525f564541551653","status":"pending"}
14:01:25 [14:01:25.361]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9d22aa22a8112500 type=BASE_PARITY startedAt=1750255284307 older agent has taken job
{"provingJobId":"1:BASE_PARITY:9d22aa22a8112500"}
14:01:25 [14:01:25.362]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9d22aa22a8112500 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:9d22aa22a8112500"}
14:01:25 [14:01:25.364]
INFO:
prover-client:proving-broker Proving Broker started
14:01:25 [14:01:25.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba6112db7a7eb9f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba6112db7a7eb9f2"}
14:01:25 [14:01:25.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:306273362077f049 epochNumber=2
{"provingJobId":"1:BASE_PARITY:306273362077f049"}
14:01:26 [14:01:26.367]
INFO:
prover-client:proving-broker Proving Broker started
14:01:26 [14:01:26.367]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ba6112db7a7eb9f2 settled=false
{"provingJobId":"1:BASE_PARITY:ba6112db7a7eb9f2","status":"pending"}
14:01:26 [14:01:26.368]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:306273362077f049 settled=false
{"provingJobId":"1:BASE_PARITY:306273362077f049","status":"pending"}
14:01:26 [14:01:26.368]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ba6112db7a7eb9f2 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:ba6112db7a7eb9f2"}
14:01:26 [14:01:26.370]
INFO:
prover-client:proving-broker Proving Broker started
14:01:26 [14:01:26.371]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ebd61a12053dd8cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:ebd61a12053dd8cb"}
14:01:26 [14:01:26.371]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8292c2c6daf89a3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f8292c2c6daf89a3"}
14:01:27 [14:01:27.372]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.372]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ebd61a12053dd8cb settled=false
{"provingJobId":"1:BASE_PARITY:ebd61a12053dd8cb","status":"pending"}
14:01:27 [14:01:27.373]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f8292c2c6daf89a3 settled=false
{"provingJobId":"1:BASE_PARITY:f8292c2c6daf89a3","status":"pending"}
14:01:27 [14:01:27.374]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ebd61a12053dd8cb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ebd61a12053dd8cb"}
14:01:27 [14:01:27.374]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ebd61a12053dd8cb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ebd61a12053dd8cb"}
14:01:27 [14:01:27.376]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.377]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68eda9143aa7e68a epochNumber=1
{"provingJobId":"1:BASE_PARITY:68eda9143aa7e68a"}
14:01:27 [14:01:27.378]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9cc87f0e47de51f3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9cc87f0e47de51f3"}
14:01:27 [14:01:27.379]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:68eda9143aa7e68a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:68eda9143aa7e68a"}
14:01:27 [14:01:27.379]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9cc87f0e47de51f3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9cc87f0e47de51f3"}
14:01:27 [14:01:27.381]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.381]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed5f393907c236fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed5f393907c236fb"}
14:01:27 [14:01:27.382]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d68948533c623ca epochNumber=2
{"provingJobId":"1:BASE_PARITY:5d68948533c623ca"}
14:01:27 [14:01:27.382]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ed5f393907c236fb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ed5f393907c236fb"}
14:01:27 [14:01:27.382]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ed5f393907c236fb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ed5f393907c236fb"}
14:01:27 [14:01:27.383]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5d68948533c623ca type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5d68948533c623ca"}
14:01:27 [14:01:27.384]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.384]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:540893c06eacbae9 err=test error
{"provingJoId":"1:BASE_PARITY:540893c06eacbae9"}
14:01:27 [14:01:27.386]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.386]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:038b89481ed1d7af not found
{"provingJobId":"1:BASE_PARITY:038b89481ed1d7af"}
14:01:27 [14:01:27.388]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.388]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c72bc12a85e9b1fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:c72bc12a85e9b1fe"}
14:01:27 [14:01:27.389]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.390]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:23c942a2710fb323 epochNumber=1
{"provingJobId":"1:BASE_PARITY:23c942a2710fb323"}
14:01:27 [14:01:27.491]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:23c942a2710fb323 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:23c942a2710fb323"}
14:01:27 [14:01:27.499]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:779dd32b1532f503 epochNumber=1
{"provingJobId":"1:BASE_PARITY:779dd32b1532f503"}
14:01:27 [14:01:27.601]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:779dd32b1532f503 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:779dd32b1532f503"}
14:01:27 [14:01:27.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34755887aff6a447 epochNumber=10
{"provingJobId":"1:BASE_PARITY:34755887aff6a447"}
14:01:27 [14:01:27.628]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:27 [14:01:27.736]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.737]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f1e57fd5eb02cda7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f1e57fd5eb02cda7"}
14:01:27 [14:01:27.890]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f1e57fd5eb02cda7 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f1e57fd5eb02cda7"}
14:01:27 [14:01:27.942]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.943]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83b18544103b0fd5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:83b18544103b0fd5"}
14:01:27 [14:01:27.944]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:83b18544103b0fd5 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:83b18544103b0fd5"}
14:01:27 [14:01:27.945]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a573a9dc3433370 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a573a9dc3433370"}
14:01:27 [14:01:27.947]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5a573a9dc3433370 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:5a573a9dc3433370"}
14:01:27 [14:01:27.949]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5a573a9dc3433370 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:5a573a9dc3433370"}
14:01:27 [14:01:27.951]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.952]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a2df2dd9b9bb6137 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a2df2dd9b9bb6137"}
14:01:27 [14:01:27.952]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a2df2dd9b9bb6137 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a2df2dd9b9bb6137"}
14:01:27 [14:01:27.954]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.954]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5542ca87f3d59e32 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5542ca87f3d59e32"}
14:01:27 [14:01:27.955]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5542ca87f3d59e32 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:5542ca87f3d59e32"}
14:01:27 [14:01:27.956]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de9856d861e95f91 epochNumber=3
{"provingJobId":"1:BASE_PARITY:de9856d861e95f91"}
14:01:27 [14:01:27.980]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:27 [14:01:27.982]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:5542ca87f3d59e32 err=test error
{"provingJoId":"1:BASE_PARITY:5542ca87f3d59e32"}
14:01:27 [14:01:27.984]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.984]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9f052691982f1671 settled=false
{"provingJobId":"1:BASE_PARITY:9f052691982f1671","status":"pending"}
14:01:27 [14:01:27.985]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e12f57ab061be7d2 settled=false
{"provingJobId":"1:BASE_PARITY:e12f57ab061be7d2","status":"pending"}
14:01:27 [14:01:27.988]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.988]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8370da474d9fd445 settled=true
{"provingJobId":"1:BASE_PARITY:8370da474d9fd445","status":"fulfilled"}
14:01:27 [14:01:27.989]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:196dfb53ca2c743f settled=true
{"provingJobId":"2:BASE_PARITY:196dfb53ca2c743f","status":"fulfilled"}
14:01:27 [14:01:27.990]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.991]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:650a2901976f0305 settled=true
{"provingJobId":"1:BASE_PARITY:650a2901976f0305","status":"fulfilled"}
14:01:27 [14:01:27.991]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c36005690983b273 settled=false
{"provingJobId":"1:BASE_PARITY:c36005690983b273","status":"pending"}
14:01:27 [14:01:27.993]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.994]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c3ad3fda28e5020 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c3ad3fda28e5020"}
14:01:27 [14:01:27.996]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:27.996]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5354a04dc1912000 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5354a04dc1912000"}
14:01:27 [14:01:27.997]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:5354a04dc1912000: Error: db error: Error: db error
14:01:27 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:5354a04dc1912000"}
14:01:27 [14:01:27.999]
INFO:
prover-client:proving-broker Proving Broker started
14:01:27 [14:01:28.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84ef41cda7350e9e epochNumber=1
{"provingJobId":"1:BASE_PARITY:84ef41cda7350e9e"}
14:01:28 [14:01:28.001]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:84ef41cda7350e9e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:84ef41cda7350e9e"}
14:01:28 [14:01:28.001]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:84ef41cda7350e9e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:84ef41cda7350e9e"}
14:01:28 [14:01:28.003]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.003]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0672b7c5e318399 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0672b7c5e318399"}
14:01:28 [14:01:28.004]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e0672b7c5e318399 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e0672b7c5e318399"}
14:01:28 [14:01:28.004]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e0672b7c5e318399 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e0672b7c5e318399"}
14:01:28 [14:01:28.005]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:e0672b7c5e318399: Error: db error
14:01:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:e0672b7c5e318399"}
14:01:28 [14:01:28.007]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.008]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:772fd7d6f287f38d epochNumber=1
{"provingJobId":"1:BASE_PARITY:772fd7d6f287f38d"}
14:01:28 [14:01:28.008]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:772fd7d6f287f38d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:772fd7d6f287f38d"}
14:01:28 [14:01:28.009]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:772fd7d6f287f38d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:772fd7d6f287f38d"}
14:01:28 [14:01:28.011]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a81ca4507cbf8567 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a81ca4507cbf8567"}
14:01:28 [14:01:28.012]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a81ca4507cbf8567 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a81ca4507cbf8567"}
14:01:28 [14:01:28.013]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a81ca4507cbf8567 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a81ca4507cbf8567"}
14:01:28 [14:01:28.013]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a81ca4507cbf8567 jobErr=test error: Error: db error
14:01:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:a81ca4507cbf8567"}
14:01:28 [14:01:28.015]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.015]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3138ff5b9de88c0a not found
{"provingJobId":"1:BASE_PARITY:3138ff5b9de88c0a"}
14:01:28 [14:01:28.016]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.017]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e86c03a37596e806 err=test error
{"provingJoId":"1:BASE_PARITY:e86c03a37596e806"}
14:01:28 [14:01:28.018]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7321fd130d40e8eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:7321fd130d40e8eb"}
14:01:28 [14:01:28.030]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7321fd130d40e8eb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7321fd130d40e8eb"}
14:01:28 [14:01:28.030]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7321fd130d40e8eb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7321fd130d40e8eb"}
14:01:28 [14:01:28.041]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:e81942a1196b0cec epochNumber=2
{"provingJobId":"2:BASE_PARITY:e81942a1196b0cec"}
14:01:28 [14:01:28.042]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:e81942a1196b0cec type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:e81942a1196b0cec"}
14:01:28 [14:01:28.042]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:e81942a1196b0cec type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:e81942a1196b0cec"}
14:01:28 [14:01:28.103]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:8a70d211961f1d8a epochNumber=3
{"provingJobId":"3:BASE_PARITY:8a70d211961f1d8a"}
14:01:28 [14:01:28.120]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:28 [14:01:28.165]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:decb52b88828131d epochNumber=4
{"provingJobId":"4:BASE_PARITY:decb52b88828131d"}
14:01:28 [14:01:28.171]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:28 [14:01:28.227]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:42d7fa506dbe325d epochNumber=5
{"provingJobId":"5:BASE_PARITY:42d7fa506dbe325d"}
14:01:28 [14:01:28.248]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:28 [14:01:28.283]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a642609183328b0 epochNumber=42
{"provingJobId":"1:BASE_PARITY:3a642609183328b0"}
14:01:28 [14:01:28.285]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testxFiJ6l/42 with map size 1073741824
14:01:28 [14:01:28.291]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.312]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:95bdef892dd4634b epochNumber=1
{"provingJobId":"1:BASE_PARITY:95bdef892dd4634b"}
14:01:28 [14:01:28.317]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.318]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2db25696375f1fe6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2db25696375f1fe6"}
14:01:28 [14:01:28.319]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEm0EXb/1 with map size 1073741824
14:01:28 [14:01:28.321]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:151fe792046b4b48 epochNumber=1
{"provingJobId":"1:BASE_PARITY:151fe792046b4b48"}
14:01:28 [14:01:28.344]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.347]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e71053e493fd3942 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e71053e493fd3942"}
14:01:28 [14:01:28.348]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1ANr05/1 with map size 1073741824
14:01:28 [14:01:28.356]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.380]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e71053e493fd3942 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e71053e493fd3942"}
14:01:28 [14:01:28.385]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.386]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67a6601682d49083 epochNumber=1
{"provingJobId":"1:BASE_PARITY:67a6601682d49083"}
14:01:28 [14:01:28.387]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiK0wMF/1 with map size 1073741824
14:01:28 [14:01:28.390]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.409]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:67a6601682d49083 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:67a6601682d49083"}
14:01:28 [14:01:28.409]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:67a6601682d49083 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:67a6601682d49083"}
14:01:28 [14:01:28.416]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:67a6601682d49083 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:67a6601682d49083"}
14:01:28 [14:01:28.421]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.421]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43f281927a9531e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43f281927a9531e2"}
14:01:28 [14:01:28.422]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVbJao8/1 with map size 1073741824
14:01:28 [14:01:28.425]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.457]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.461]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.461]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb4265652fc1110e epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb4265652fc1110e"}
14:01:28 [14:01:28.462]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhjRZd1/1 with map size 1073741824
14:01:28 [14:01:28.463]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.486]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:bb4265652fc1110e
{"provingJobId":"1:BASE_PARITY:bb4265652fc1110e"}
14:01:28 [14:01:28.486]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bb4265652fc1110e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bb4265652fc1110e"}
14:01:28 [14:01:28.487]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bb4265652fc1110e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:bb4265652fc1110e"}
14:01:28 [14:01:28.499]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06bc86654f1b57d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:06bc86654f1b57d5"}
14:01:28 [14:01:28.501]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testg4b925/1 with map size 1073741824
14:01:28 [14:01:28.503]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.520]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:06bc86654f1b57d5
{"provingJobId":"1:BASE_PARITY:06bc86654f1b57d5"}
14:01:28 [14:01:28.521]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:06bc86654f1b57d5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:06bc86654f1b57d5"}
14:01:28 [14:01:28.531]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.531]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a6441a078859094 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a6441a078859094"}
14:01:28 [14:01:28.532]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDhWktM/1 with map size 1073741824
14:01:28 [14:01:28.540]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.562]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0a6441a078859094 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0a6441a078859094"}
14:01:28 [14:01:28.562]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0a6441a078859094 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0a6441a078859094"}
14:01:28 [14:01:28.571]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.572]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb1cd6bc31885dbc epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb1cd6bc31885dbc"}
14:01:28 [14:01:28.572]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQ9afEH/1 with map size 1073741824
14:01:28 [14:01:28.574]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.590]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb1cd6bc31885dbc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:eb1cd6bc31885dbc"}
14:01:28 [14:01:28.591]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:eb1cd6bc31885dbc type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:eb1cd6bc31885dbc"}
14:01:28 [14:01:28.599]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.600]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b59eda297f53070 epochNumber=0
{"provingJobId":"1:BASE_PARITY:9b59eda297f53070"}
14:01:28 [14:01:28.601]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testJ59cnc/0 with map size 1073741824
14:01:28 [14:01:28.603]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.622]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9b59eda297f53070 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9b59eda297f53070"}
14:01:28 [14:01:28.623]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:332303ba53c207fc epochNumber=0
{"provingJobId":"1:BASE_PARITY:332303ba53c207fc"}
14:01:28 [14:01:28.624]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9b59eda297f53070 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9b59eda297f53070"}
14:01:28 [14:01:28.624]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aeef3a36c38edb28 epochNumber=0
{"provingJobId":"1:BASE_PARITY:aeef3a36c38edb28"}
14:01:28 [14:01:28.640]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.642]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.642]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a99518e54d45334 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6a99518e54d45334"}
14:01:28 [14:01:28.643]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDMz5bg/2 with map size 1073741824
14:01:28 [14:01:28.651]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.673]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50368a078c803e97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:50368a078c803e97"}
14:01:28 [14:01:28.674]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDMz5bg/1 with map size 1073741824
14:01:28 [14:01:28.677]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.698]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.699]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c040c314a7d98cee epochNumber=1
{"provingJobId":"1:BASE_PARITY:c040c314a7d98cee"}
14:01:28 [14:01:28.699]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2PoMQI/1 with map size 1073741824
14:01:28 [14:01:28.701]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.722]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.723]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d9e179f6b86104e4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d9e179f6b86104e4"}
14:01:28 [14:01:28.724]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3QMGGI/1 with map size 1073741824
14:01:28 [14:01:28.726]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.743]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f9ebeaaea732202 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f9ebeaaea732202"}
14:01:28 [14:01:28.754]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a2657aec556abb63 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a2657aec556abb63"}
14:01:28 [14:01:28.756]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test3QMGGI/2 with map size 1073741824
14:01:28 [14:01:28.758]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f57e5b798bdaf06c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f57e5b798bdaf06c"}
14:01:28 [14:01:28.784]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e8c5acb00af944f epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e8c5acb00af944f"}
14:01:28 [14:01:28.786]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGKxxus/1 with map size 1073741824
14:01:28 [14:01:28.787]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5cd1e70bdd646ad7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5cd1e70bdd646ad7"}
14:01:28 [14:01:28.810]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:975bce5bda4fa8c6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:975bce5bda4fa8c6"}
14:01:28 [14:01:28.811]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGKxxus/2 with map size 1073741824
14:01:28 [14:01:28.813]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.829]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f79043f008b0fc18 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f79043f008b0fc18"}
14:01:28 [14:01:28.838]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.838]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a4b4546b0efbda7f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a4b4546b0efbda7f"}
14:01:28 [14:01:28.839]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCx4aOu/1 with map size 1073741824
14:01:28 [14:01:28.841]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.861]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ed7f63d0caa2529 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ed7f63d0caa2529"}
14:01:28 [14:01:28.868]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff662c8ac6866222 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ff662c8ac6866222"}
14:01:28 [14:01:28.870]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCx4aOu/2 with map size 1073741824
14:01:28 [14:01:28.873]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.891]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2444c890a45bc303 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2444c890a45bc303"}
14:01:28 [14:01:28.902]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.903]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:777d7d1eb13138c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:777d7d1eb13138c2"}
14:01:28 [14:01:28.903]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzAEVtP/1 with map size 1073741824
14:01:28 [14:01:28.906]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.925]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5dd3f46b1aff0b68 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5dd3f46b1aff0b68"}
14:01:28 [14:01:28.931]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:777d7d1eb13138c2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:777d7d1eb13138c2"}
14:01:28 [14:01:28.943]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc64b8d5f22de741 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc64b8d5f22de741"}
14:01:28 [14:01:28.946]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmg1uWx/1 with map size 1073741824
14:01:28 [14:01:28.950]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:28 [14:01:28.970]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e8d79c33036f741 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e8d79c33036f741"}
14:01:28 [14:01:28.976]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fc64b8d5f22de741 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:fc64b8d5f22de741"}
14:01:28 [14:01:28.987]
INFO:
prover-client:proving-broker Proving Broker started
14:01:28 [14:01:28.988]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:085599fa315c9743 epochNumber=1
{"provingJobId":"1:BASE_PARITY:085599fa315c9743"}
14:01:28 [14:01:28.989]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test23rqiX/1 with map size 1073741824
14:01:28 [14:01:28.992]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:29 [14:01:29.008]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de0764bdb6f04b32 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de0764bdb6f04b32"}
14:01:29 [14:01:29.014]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:085599fa315c9743 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:085599fa315c9743"}
14:01:29 [14:01:29.020]
INFO:
prover-client:proving-broker Proving Broker started
14:01:29 [14:01:29.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95d64583cc2a8d5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:95d64583cc2a8d5d"}
14:01:29 [14:01:29.021]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbwkVNk/1 with map size 1073741824
14:01:29 [14:01:29.023]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:29 [14:01:29.040]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:95d64583cc2a8d5d
{"provingJobId":"1:BASE_PARITY:95d64583cc2a8d5d"}
14:01:29 [14:01:29.040]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:95d64583cc2a8d5d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:95d64583cc2a8d5d"}
14:01:29 [14:01:29.047]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff0768b18ae35529 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff0768b18ae35529"}
14:01:29 [14:01:29.053]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:95d64583cc2a8d5d has already been completed
{"provingJobId":"1:BASE_PARITY:95d64583cc2a8d5d"}
14:01:29 [14:01:29.057]
INFO:
prover-client:proving-broker Proving Broker started
14:01:29 [14:01:29.058]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b3723073e8c9160 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b3723073e8c9160"}
14:01:29 [14:01:29.059]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjnbTfq/1 with map size 1073741824
14:01:29 [14:01:29.062]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:29 [14:01:29.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c603aad519eef508 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c603aad519eef508"}
14:01:29 [14:01:29.080]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjnbTfq/2 with map size 1073741824
14:01:29 [14:01:29.082]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:30 [14:01:30.150]
INFO:
prover-client:proving-broker Proving Broker started
14:01:30 [14:01:30.157]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2b3723073e8c9160 settled=false
{"provingJobId":"1:BASE_PARITY:2b3723073e8c9160","status":"pending"}
14:01:30 [14:01:30.159]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c603aad519eef508 settled=false
{"provingJobId":"1:BASE_PARITY:c603aad519eef508","status":"pending"}
14:01:30 [14:01:30.160]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2b3723073e8c9160 type=BASE_PARITY startedAt=1750255289099 older agent has taken job
{"provingJobId":"1:BASE_PARITY:2b3723073e8c9160"}
14:01:30 [14:01:30.161]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2b3723073e8c9160 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:2b3723073e8c9160"}
14:01:30 [14:01:30.167]
INFO:
prover-client:proving-broker Proving Broker started
14:01:30 [14:01:30.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10ba28fa1ae64b08 epochNumber=1
{"provingJobId":"1:BASE_PARITY:10ba28fa1ae64b08"}
14:01:30 [14:01:30.170]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlEBK1A/1 with map size 1073741824
14:01:30 [14:01:30.172]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:30 [14:01:30.192]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c0820708096cf71 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7c0820708096cf71"}
14:01:30 [14:01:30.194]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlEBK1A/2 with map size 1073741824
14:01:30 [14:01:30.199]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:31 [14:01:31.221]
INFO:
prover-client:proving-broker Proving Broker started
14:01:31 [14:01:31.223]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:10ba28fa1ae64b08 settled=false
{"provingJobId":"1:BASE_PARITY:10ba28fa1ae64b08","status":"pending"}
14:01:31 [14:01:31.225]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7c0820708096cf71 settled=false
{"provingJobId":"1:BASE_PARITY:7c0820708096cf71","status":"pending"}
14:01:31 [14:01:31.226]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:10ba28fa1ae64b08 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:10ba28fa1ae64b08"}
14:01:31 [14:01:31.229]
INFO:
prover-client:proving-broker Proving Broker started
14:01:31 [14:01:31.230]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59101be7974a8a9d epochNumber=1
{"provingJobId":"1:BASE_PARITY:59101be7974a8a9d"}
14:01:31 [14:01:31.230]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqQSbgT/1 with map size 1073741824
14:01:31 [14:01:31.232]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:31 [14:01:31.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b1ce9c491cfb69b epochNumber=2
{"provingJobId":"1:BASE_PARITY:5b1ce9c491cfb69b"}
14:01:31 [14:01:31.251]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqQSbgT/2 with map size 1073741824
14:01:31 [14:01:31.259]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:32 [14:01:32.279]
INFO:
prover-client:proving-broker Proving Broker started
14:01:32 [14:01:32.281]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:59101be7974a8a9d settled=false
{"provingJobId":"1:BASE_PARITY:59101be7974a8a9d","status":"pending"}
14:01:32 [14:01:32.283]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5b1ce9c491cfb69b settled=false
{"provingJobId":"1:BASE_PARITY:5b1ce9c491cfb69b","status":"pending"}
14:01:32 [14:01:32.284]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:59101be7974a8a9d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:59101be7974a8a9d"}
14:01:32 [14:01:32.285]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:59101be7974a8a9d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:59101be7974a8a9d"}
14:01:32 [14:01:32.296]
INFO:
prover-client:proving-broker Proving Broker started
14:01:32 [14:01:32.298]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be053c1440cdd4b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:be053c1440cdd4b3"}
14:01:32 [14:01:32.298]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testynTMNa/1 with map size 1073741824
14:01:32 [14:01:32.300]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:32 [14:01:32.319]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7fca653a8eb20dce epochNumber=2
{"provingJobId":"1:BASE_PARITY:7fca653a8eb20dce"}
14:01:32 [14:01:32.320]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testynTMNa/2 with map size 1073741824
14:01:32 [14:01:32.322]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:32 [14:01:32.343]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:be053c1440cdd4b3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:be053c1440cdd4b3"}
14:01:32 [14:01:32.350]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7fca653a8eb20dce type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7fca653a8eb20dce"}
14:01:32 [14:01:32.361]
INFO:
prover-client:proving-broker Proving Broker started
14:01:32 [14:01:32.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22654e375b786f42 epochNumber=1
{"provingJobId":"1:BASE_PARITY:22654e375b786f42"}
14:01:32 [14:01:32.363]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxuewTW/1 with map size 1073741824
14:01:32 [14:01:32.365]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:32 [14:01:32.389]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87689029b86e28b4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:87689029b86e28b4"}
14:01:32 [14:01:32.390]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testxuewTW/2 with map size 1073741824
14:01:32 [14:01:32.393]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:32 [14:01:32.417]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:22654e375b786f42 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:22654e375b786f42"}
14:01:32 [14:01:32.418]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:22654e375b786f42 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:22654e375b786f42"}
14:01:32 [14:01:32.425]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:87689029b86e28b4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:87689029b86e28b4"}
14:01:32 [14:01:32.442]
INFO:
prover-client:proving-broker Proving Broker started
14:01:32 [14:01:32.443]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:71e3680b87e27522 err=test error
{"provingJoId":"1:BASE_PARITY:71e3680b87e27522"}
14:01:32 [14:01:32.446]
INFO:
prover-client:proving-broker Proving Broker started
14:01:32 [14:01:32.448]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f35e32726eef207c not found
{"provingJobId":"1:BASE_PARITY:f35e32726eef207c"}
14:01:32 [14:01:32.453]
INFO:
prover-client:proving-broker Proving Broker started
14:01:32 [14:01:32.454]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aefe6ca4ea358c1d epochNumber=1
{"provingJobId":"1:BASE_PARITY:aefe6ca4ea358c1d"}
14:01:32 [14:01:32.456]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqNKMmN/1 with map size 1073741824
14:01:32 [14:01:32.460]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:32 [14:01:32.494]
INFO:
prover-client:proving-broker Proving Broker started
14:01:32 [14:01:32.495]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94e4fab39e3478dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:94e4fab39e3478dc"}
14:01:32 [14:01:32.497]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkm4gg0/1 with map size 1073741824
14:01:32 [14:01:32.502]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:32 [14:01:32.624]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:94e4fab39e3478dc timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:94e4fab39e3478dc"}
14:01:32 [14:01:32.635]
INFO:
prover-client:proving-broker Proving Broker started
14:01:32 [14:01:32.636]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e63d350743330f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e63d350743330f3"}
14:01:32 [14:01:32.637]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiW1ti4/1 with map size 1073741824
14:01:32 [14:01:32.641]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:32 [14:01:32.788]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6e63d350743330f3 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6e63d350743330f3"}
14:01:32 [14:01:32.793]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98030d5a74ed505d epochNumber=10
{"provingJobId":"1:BASE_PARITY:98030d5a74ed505d"}
14:01:32 [14:01:32.794]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testiW1ti4/10 with map size 1073741824
14:01:32 [14:01:32.798]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:32 [14:01:32.839]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:32 [14:01:32.840]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:01:32 [14:01:32.945]
INFO:
prover-client:proving-broker Proving Broker started
14:01:32 [14:01:32.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6dcf864a9c5c8a72 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6dcf864a9c5c8a72"}
14:01:32 [14:01:32.947]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjzaPKg/1 with map size 1073741824
14:01:32 [14:01:32.951]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.148]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6dcf864a9c5c8a72 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6dcf864a9c5c8a72"}
14:01:33 [14:01:33.182]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.183]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e99afd33e99a901 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e99afd33e99a901"}
14:01:33 [14:01:33.184]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjLOP5G/1 with map size 1073741824
14:01:33 [14:01:33.187]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.207]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1e99afd33e99a901 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1e99afd33e99a901"}
14:01:33 [14:01:33.211]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.212]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6a60981b6d3fad5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6a60981b6d3fad5"}
14:01:33 [14:01:33.213]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5e9CqV/1 with map size 1073741824
14:01:33 [14:01:33.215]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.235]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a6a60981b6d3fad5 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a6a60981b6d3fad5"}
14:01:33 [14:01:33.236]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a6a60981b6d3fad5 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:a6a60981b6d3fad5"}
14:01:33 [14:01:33.248]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.249]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c594c426772b359d epochNumber=1
{"provingJobId":"1:BASE_PARITY:c594c426772b359d"}
14:01:33 [14:01:33.250]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVbMgbr/1 with map size 1073741824
14:01:33 [14:01:33.253]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.272]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c594c426772b359d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c594c426772b359d"}
14:01:33 [14:01:33.284]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.285]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a37909f84ae3935e epochNumber=1
{"provingJobId":"1:BASE_PARITY:a37909f84ae3935e"}
14:01:33 [14:01:33.287]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbhrR4C/1 with map size 1073741824
14:01:33 [14:01:33.289]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.312]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a37909f84ae3935e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a37909f84ae3935e"}
14:01:33 [14:01:33.313]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:485c03a06a327490 epochNumber=3
{"provingJobId":"1:BASE_PARITY:485c03a06a327490"}
14:01:33 [14:01:33.314]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testbhrR4C/3 with map size 1073741824
14:01:33 [14:01:33.318]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.359]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:33 [14:01:33.360]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:01:33 [14:01:33.365]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a37909f84ae3935e err=test error
{"provingJoId":"1:BASE_PARITY:a37909f84ae3935e"}
14:01:33 [14:01:33.373]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuNvmaF/1 with map size 1073741824
14:01:33 [14:01:33.376]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.399]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testuNvmaF/2 with map size 1073741824
14:01:33 [14:01:33.401]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.421]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.424]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f2d627557d161387 settled=false
{"provingJobId":"1:BASE_PARITY:f2d627557d161387","status":"pending"}
14:01:33 [14:01:33.426]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:16d07e4684cccd92 settled=false
{"provingJobId":"1:BASE_PARITY:16d07e4684cccd92","status":"pending"}
14:01:33 [14:01:33.436]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testySqZkw/1 with map size 1073741824
14:01:33 [14:01:33.439]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.461]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testySqZkw/2 with map size 1073741824
14:01:33 [14:01:33.467]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.498]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.500]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7dea86fcaae75bf6 settled=true
{"provingJobId":"1:BASE_PARITY:7dea86fcaae75bf6","status":"fulfilled"}
14:01:33 [14:01:33.502]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:59cf47499ea9806a settled=true
{"provingJobId":"2:BASE_PARITY:59cf47499ea9806a","status":"fulfilled"}
14:01:33 [14:01:33.509]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQQSWsi/1 with map size 1073741824
14:01:33 [14:01:33.512]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.548]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQQSWsi/2 with map size 1073741824
14:01:33 [14:01:33.552]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.570]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.572]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e269573cb67b470e settled=true
{"provingJobId":"1:BASE_PARITY:e269573cb67b470e","status":"fulfilled"}
14:01:33 [14:01:33.574]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:eb57d6772100886f settled=false
{"provingJobId":"1:BASE_PARITY:eb57d6772100886f","status":"pending"}
14:01:33 [14:01:33.580]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.581]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32a04037280ae78e epochNumber=1
{"provingJobId":"1:BASE_PARITY:32a04037280ae78e"}
14:01:33 [14:01:33.582]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwjDZfz/1 with map size 1073741824
14:01:33 [14:01:33.586]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.616]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.617]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2688aeb1981278dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:2688aeb1981278dd"}
14:01:33 [14:01:33.618]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:2688aeb1981278dd: Error: db error: Error: db error
14:01:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:2688aeb1981278dd"}
14:01:33 [14:01:33.622]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.623]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb0210d043e4d8b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb0210d043e4d8b9"}
14:01:33 [14:01:33.623]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvUb5Em/1 with map size 1073741824
14:01:33 [14:01:33.625]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.643]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bb0210d043e4d8b9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bb0210d043e4d8b9"}
14:01:33 [14:01:33.643]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bb0210d043e4d8b9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bb0210d043e4d8b9"}
14:01:33 [14:01:33.656]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.657]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ac174478a822b05 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ac174478a822b05"}
14:01:33 [14:01:33.657]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOd5b93/1 with map size 1073741824
14:01:33 [14:01:33.661]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.683]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9ac174478a822b05 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9ac174478a822b05"}
14:01:33 [14:01:33.683]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9ac174478a822b05 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9ac174478a822b05"}
14:01:33 [14:01:33.684]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:9ac174478a822b05: Error: db error
14:01:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:9ac174478a822b05"}
14:01:33 [14:01:33.691]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.691]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0566517bc99f2f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0566517bc99f2f2"}
14:01:33 [14:01:33.692]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7g1DdI/1 with map size 1073741824
14:01:33 [14:01:33.695]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.727]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e0566517bc99f2f2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e0566517bc99f2f2"}
14:01:33 [14:01:33.728]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e0566517bc99f2f2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e0566517bc99f2f2"}
14:01:33 [14:01:33.739]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b3c5d136574bb4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b3c5d136574bb4b"}
14:01:33 [14:01:33.742]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9ZbUxZ/1 with map size 1073741824
14:01:33 [14:01:33.744]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.774]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0b3c5d136574bb4b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0b3c5d136574bb4b"}
14:01:33 [14:01:33.774]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0b3c5d136574bb4b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0b3c5d136574bb4b"}
14:01:33 [14:01:33.775]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:0b3c5d136574bb4b jobErr=test error: Error: db error
14:01:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:0b3c5d136574bb4b"}
14:01:33 [14:01:33.781]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.781]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:56ad6d9b7db724e3 not found
{"provingJobId":"1:BASE_PARITY:56ad6d9b7db724e3"}
14:01:33 [14:01:33.786]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.786]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8f8d5d67aef2c8ee err=test error
{"provingJoId":"1:BASE_PARITY:8f8d5d67aef2c8ee"}
14:01:33 [14:01:33.790]
INFO:
prover-client:proving-broker Proving Broker started
14:01:33 [14:01:33.800]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57a546fea787c275 epochNumber=1
{"provingJobId":"1:BASE_PARITY:57a546fea787c275"}
14:01:33 [14:01:33.802]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJ0RSXB/1 with map size 1073741824
14:01:33 [14:01:33.806]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.826]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:57a546fea787c275 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:57a546fea787c275"}
14:01:33 [14:01:33.827]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:57a546fea787c275 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:57a546fea787c275"}
14:01:33 [14:01:33.843]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:ffdae9f11621da9c epochNumber=2
{"provingJobId":"2:BASE_PARITY:ffdae9f11621da9c"}
14:01:33 [14:01:33.849]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJ0RSXB/2 with map size 1073741824
14:01:33 [14:01:33.857]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.877]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:ffdae9f11621da9c type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:ffdae9f11621da9c"}
14:01:33 [14:01:33.877]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:ffdae9f11621da9c type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:ffdae9f11621da9c"}
14:01:33 [14:01:33.943]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:6cdf8ed89b7a1c68 epochNumber=3
{"provingJobId":"3:BASE_PARITY:6cdf8ed89b7a1c68"}
14:01:33 [14:01:33.945]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testJ0RSXB/3 with map size 1073741824
14:01:33 [14:01:33.949]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:33 [14:01:33.994]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:33 [14:01:33.994]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:01:34 [14:01:34.034]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:f10e1c904426c29e epochNumber=4
{"provingJobId":"4:BASE_PARITY:f10e1c904426c29e"}
14:01:34 [14:01:34.035]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testJ0RSXB/4 with map size 1073741824
14:01:34 [14:01:34.037]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:34 [14:01:34.075]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:34 [14:01:34.076]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
14:01:34 [14:01:34.123]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:51ff693052f75313 epochNumber=5
{"provingJobId":"5:BASE_PARITY:51ff693052f75313"}
14:01:34 [14:01:34.125]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testJ0RSXB/5 with map size 1073741824
14:01:34 [14:01:34.129]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:01:34 [14:01:34.154]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:01:34 [14:01:34.155]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
14:01:34
PASS src/proving_broker/proving_broker.test.ts (
13.186 s)
14:01:34 ProvingBroker
14:01:34 Producer API
14:01:34
✓ refuses stale jobs (42 ms)
14:01:34
✓ enqueues jobs (5 ms)
14:01:34
✓ ignores duplicate jobs (5 ms)
14:01:34
✓ reports correct status when enqueuing repeat jobs (5 ms)
14:01:34
✓ throws an error in case of duplicate job IDs (29 ms)
14:01:34
✓ returns not-found status for non-existing jobs (2 ms)
14:01:34
✓ cancels jobs in queue (4 ms)
14:01:34
✓ cancels jobs in-progress (3 ms)
14:01:34
✓ returns job result if successful (2 ms)
14:01:34
✓ returns job error if failed (2 ms)
14:01:34
✓ correctly returns job status for concurrent writes (3 ms)
14:01:34
✓ refuses stale jobs (34 ms)
14:01:34
✓ enqueues jobs (27 ms)
14:01:34
✓ ignores duplicate jobs (40 ms)
14:01:34
✓ reports correct status when enqueuing repeat jobs (35 ms)
14:01:34
✓ throws an error in case of duplicate job IDs (37 ms)
14:01:34
✓ returns not-found status for non-existing jobs (4 ms)
14:01:34
✓ cancels jobs in queue (37 ms)
14:01:34
✓ cancels jobs in-progress (31 ms)
14:01:34
✓ returns job result if successful (42 ms)
14:01:34
✓ returns job error if failed (27 ms)
14:01:34
✓ correctly returns job status for concurrent writes (40 ms)
14:01:34 Consumer API
14:01:34
✓ returns undefined if no jobs are available (18 ms)
14:01:34
✓ returns jobs in priority order (2 ms)
14:01:34
✓ returns undefined if no jobs are available for the given allowList (1 ms)
14:01:34
✓ returns a job if it is in the allowList (4 ms)
14:01:34
✓ returns the most important job if it is in the allowList (5 ms)
14:01:34
✓ returns any job if filter is empty (3 ms)
14:01:34
✓ returns a new job when reporting job success (3 ms)
14:01:34
✓ returns a new job when reporting permanent error (3 ms)
14:01:34
✓ returns a new job when reporting retry-able error (2 ms)
14:01:34
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
14:01:34
✓ returns a new job if job is already in progress elsewhere (1057 ms)
14:01:34
✓ avoids sending the same job to a new agent after a restart (1006 ms)
14:01:34
✓ avoids sending a completed job to a new agent after a restart (1007 ms)
14:01:34
✓ tracks job result if in progress (4 ms)
14:01:34
✓ tracks job result even if job is in queue (3 ms)
14:01:34
✓ ignores reported job error if unknown job (2 ms)
14:01:34
✓ ignores job result if unknown job (2 ms)
14:01:34
✓ returns undefined if no jobs are available (2 ms)
14:01:34
✓ returns jobs in priority order (55 ms)
14:01:34
✓ returns undefined if no jobs are available for the given allowList (23 ms)
14:01:34
✓ returns a job if it is in the allowList (63 ms)
14:01:34
✓ returns the most important job if it is in the allowList (53 ms)
14:01:34
✓ returns any job if filter is empty (64 ms)
14:01:34
✓ returns a new job when reporting job success (38 ms)
14:01:34
✓ returns a new job when reporting permanent error (46 ms)
14:01:34
✓ returns a new job when reporting retry-able error (31 ms)
14:01:34
✓ returns a new job when reporting progress if current one is cancelled (38 ms)
14:01:34
✓ returns a new job if job is already in progress elsewhere (1108 ms)
14:01:34
✓ avoids sending the same job to a new agent after a restart (1063 ms)
14:01:34
✓ avoids sending a completed job to a new agent after a restart (1065 ms)
14:01:34
✓ tracks job result if in progress (64 ms)
14:01:34
✓ tracks job result even if job is in queue (79 ms)
14:01:34
✓ ignores reported job error if unknown job (5 ms)
14:01:34
✓ ignores job result if unknown job (4 ms)
14:01:34 Timeouts
14:01:34
✓ tracks in progress jobs (2 ms)
14:01:34
✓ re-enqueues jobs that time out (109 ms)
14:01:34
✓ cancel stale jobs that time out (237 ms)
14:01:34
✓ keeps the jobs in progress while it is alive (205 ms)
14:01:34
✓ tracks in progress jobs (43 ms)
14:01:34
✓ re-enqueues jobs that time out (138 ms)
14:01:34
✓ cancel stale jobs that time out (311 ms)
14:01:34
✓ keeps the jobs in progress while it is alive (235 ms)
14:01:34 Retries
14:01:34
✓ retries jobs (3 ms)
14:01:34
✓ retries up to a maximum number of times (5 ms)
14:01:34
✓ passing retry=false does not retry (3 ms)
14:01:34
✓ does not retry if job is stale (29 ms)
14:01:34
✓ retries jobs (30 ms)
14:01:34
✓ retries up to a maximum number of times (36 ms)
14:01:34
✓ passing retry=false does not retry (36 ms)
14:01:34
✓ does not retry if job is stale (87 ms)
14:01:34 Database management
14:01:34
✓ re-enqueues proof requests on start (4 ms)
14:01:34
✓ restores proof results on start (2 ms)
14:01:34
✓ only re-enqueues unfinished jobs (2 ms)
14:01:34
✓ saves job when enqueued (3 ms)
14:01:34
✓ does not retain job if database fails to save (2 ms)
14:01:34
✓ saves job result (4 ms)
14:01:34
✓ saves result even if database fails to save (4 ms)
14:01:34
✓ saves job error (4 ms)
14:01:34
✓ saves job error even if database fails to save (4 ms)
14:01:34
✓ does not save job result if job is unknown (2 ms)
14:01:34
✓ does not save job error if job is unknown (1 ms)
14:01:34
✓ cleans up old jobs periodically (263 ms)
14:01:34
✓ re-enqueues proof requests on start (61 ms)
14:01:34
✓ restores proof results on start (74 ms)
14:01:34
✓ only re-enqueues unfinished jobs (72 ms)
14:01:34
✓ saves job when enqueued (33 ms)
14:01:34
✓ does not retain job if database fails to save (7 ms)
14:01:34
✓ saves job result (33 ms)
14:01:34
✓ saves result even if database fails to save (34 ms)
14:01:34
✓ saves job error (49 ms)
14:01:34
✓ saves job error even if database fails to save (41 ms)
14:01:34
✓ does not save job result if job is unknown (4 ms)
14:01:34
✓ does not save job error if job is unknown (4 ms)
14:01:34
✓ cleans up old jobs periodically (414 ms)
14:01:34
14:01:34
Test Suites: 1 passed, 1 total
14:01:34
Tests: 96 passed, 96 total
14:01:34
Snapshots: 0 total
14:01:34
Time: 13.294 s
14:01:34
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
14:01:34
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?