Parent Log:
http://ci.aztec-labs.com/1320ea46e24a4ada
Command: fa3d426c5ffed922 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/ddabb45f79cc171e41a0a5330544775de9fdf7fd
Env: REF_NAME=gh-readonly-queue/next/pr-14983-4e03f15ca8099f25d50eddd19c4914764b84fe7b CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 13 14:37:43 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14983_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
14:37:47 [14:37:47.696]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.703]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3dc976df77b14018 epochNumber=42
{"provingJobId":"1:BASE_PARITY:3dc976df77b14018"}
14:37:47 [14:37:47.707]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:8f9df71ee4cbe128 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f9df71ee4cbe128"}
14:37:47 [14:37:47.734]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.735]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae6ab60762674d69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae6ab60762674d69"}
14:37:47 [14:37:47.736]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5059ad739e1097f7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5059ad739e1097f7"}
14:37:47 [14:37:47.738]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.738]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f8195207f469822 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7f8195207f469822"}
14:37:47 [14:37:47.739]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7f8195207f469822 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7f8195207f469822"}
14:37:47 [14:37:47.741]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ec263b581eeb3e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ec263b581eeb3e0"}
14:37:47 [14:37:47.743]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8ec263b581eeb3e0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8ec263b581eeb3e0"}
14:37:47 [14:37:47.744]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8ec263b581eeb3e0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8ec263b581eeb3e0"}
14:37:47 [14:37:47.744]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8ec263b581eeb3e0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8ec263b581eeb3e0"}
14:37:47 [14:37:47.746]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.746]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fe917c5c47043c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fe917c5c47043c5"}
14:37:47 [14:37:47.774]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.775]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f178f1387165dcdf epochNumber=1
{"provingJobId":"1:BASE_PARITY:f178f1387165dcdf"}
14:37:47 [14:37:47.777]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f178f1387165dcdf
{"provingJobId":"1:BASE_PARITY:f178f1387165dcdf"}
14:37:47 [14:37:47.777]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f178f1387165dcdf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f178f1387165dcdf"}
14:37:47 [14:37:47.778]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f178f1387165dcdf type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f178f1387165dcdf"}
14:37:47 [14:37:47.787]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.788]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31562a00d3bbb41e epochNumber=1
{"provingJobId":"1:BASE_PARITY:31562a00d3bbb41e"}
14:37:47 [14:37:47.789]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:31562a00d3bbb41e
{"provingJobId":"1:BASE_PARITY:31562a00d3bbb41e"}
14:37:47 [14:37:47.789]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:31562a00d3bbb41e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:31562a00d3bbb41e"}
14:37:47 [14:37:47.790]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9fefcd2a6f8e4c63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9fefcd2a6f8e4c63"}
14:37:47 [14:37:47.791]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9fefcd2a6f8e4c63 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9fefcd2a6f8e4c63"}
14:37:47 [14:37:47.792]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9fefcd2a6f8e4c63 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9fefcd2a6f8e4c63"}
14:37:47 [14:37:47.793]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.793]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c780031764ed3bd9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c780031764ed3bd9"}
14:37:47 [14:37:47.793]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c780031764ed3bd9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c780031764ed3bd9"}
14:37:47 [14:37:47.794]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c780031764ed3bd9 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c780031764ed3bd9"}
14:37:47 [14:37:47.794]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.795]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:926bd7ca83f9c561 epochNumber=0
{"provingJobId":"1:BASE_PARITY:926bd7ca83f9c561"}
14:37:47 [14:37:47.795]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:926bd7ca83f9c561 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:926bd7ca83f9c561"}
14:37:47 [14:37:47.795]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b5c793fd45d0f2c epochNumber=0
{"provingJobId":"1:BASE_PARITY:8b5c793fd45d0f2c"}
14:37:47 [14:37:47.795]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:926bd7ca83f9c561 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:926bd7ca83f9c561"}
14:37:47 [14:37:47.796]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7c237662ad6dfb6 epochNumber=0
{"provingJobId":"1:BASE_PARITY:b7c237662ad6dfb6"}
14:37:47 [14:37:47.797]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.798]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.798]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44366cccbd2c1a06 epochNumber=2
{"provingJobId":"1:BASE_PARITY:44366cccbd2c1a06"}
14:37:47 [14:37:47.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1bcf48f5b1b025b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1bcf48f5b1b025b"}
14:37:47 [14:37:47.800]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.800]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56ffa2e328e0f624 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56ffa2e328e0f624"}
14:37:47 [14:37:47.801]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.801]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:022b016a99a83394 epochNumber=1
{"provingJobId":"1:BASE_PARITY:022b016a99a83394"}
14:37:47 [14:37:47.801]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6cd607cdcce34e0a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6cd607cdcce34e0a"}
14:37:47 [14:37:47.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:325caa4e0359ab7d epochNumber=2
{"provingJobId":"1:BASE_PARITY:325caa4e0359ab7d"}
14:37:47 [14:37:47.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bcc74b350b96d1e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bcc74b350b96d1e5"}
14:37:47 [14:37:47.803]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.803]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d50871b51d128b04 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d50871b51d128b04"}
14:37:47 [14:37:47.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bbd6e9dad7f68d30 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bbd6e9dad7f68d30"}
14:37:47 [14:37:47.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa8d8fefdaae17d0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fa8d8fefdaae17d0"}
14:37:47 [14:37:47.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ac394e0f97be65c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ac394e0f97be65c4"}
14:37:47 [14:37:47.805]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.805]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:73868fc32376c27c epochNumber=1
{"provingJobId":"1:BASE_PARITY:73868fc32376c27c"}
14:37:47 [14:37:47.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c9de507604f1b92 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c9de507604f1b92"}
14:37:47 [14:37:47.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f41191bf54fd8c1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3f41191bf54fd8c1"}
14:37:47 [14:37:47.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d46846beeb81736f epochNumber=1
{"provingJobId":"1:BASE_PARITY:d46846beeb81736f"}
14:37:47 [14:37:47.807]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b8021bb1007d301 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5b8021bb1007d301"}
14:37:47 [14:37:47.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da566954494cd266 epochNumber=1
{"provingJobId":"1:BASE_PARITY:da566954494cd266"}
14:37:47 [14:37:47.808]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5b8021bb1007d301 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5b8021bb1007d301"}
14:37:47 [14:37:47.809]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.810]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cb6f69dd999a2f4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0cb6f69dd999a2f4"}
14:37:47 [14:37:47.810]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fac6aa3ea24ac2a epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fac6aa3ea24ac2a"}
14:37:47 [14:37:47.810]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0cb6f69dd999a2f4 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:0cb6f69dd999a2f4"}
14:37:47 [14:37:47.811]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.812]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c7615d91564d6a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c7615d91564d6a7"}
14:37:47 [14:37:47.812]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a99d8a4ac864a7f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a99d8a4ac864a7f5"}
14:37:47 [14:37:47.812]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4c7615d91564d6a7 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:4c7615d91564d6a7"}
14:37:47 [14:37:47.813]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c507c9c2f908e5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c507c9c2f908e5d"}
14:37:47 [14:37:47.814]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5c507c9c2f908e5d
{"provingJobId":"1:BASE_PARITY:5c507c9c2f908e5d"}
14:37:47 [14:37:47.814]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5c507c9c2f908e5d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5c507c9c2f908e5d"}
14:37:47 [14:37:47.815]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68db3095ec5ed58d epochNumber=1
{"provingJobId":"1:BASE_PARITY:68db3095ec5ed58d"}
14:37:47 [14:37:47.815]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5c507c9c2f908e5d has already been completed
{"provingJobId":"1:BASE_PARITY:5c507c9c2f908e5d"}
14:37:47 [14:37:47.816]
INFO:
prover-client:proving-broker Proving Broker started
14:37:47 [14:37:47.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26b2daabe6130b99 epochNumber=1
{"provingJobId":"1:BASE_PARITY:26b2daabe6130b99"}
14:37:47 [14:37:47.817]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81c3617bc7e11b5c epochNumber=2
{"provingJobId":"1:BASE_PARITY:81c3617bc7e11b5c"}
14:37:48 [14:37:48.868]
INFO:
prover-client:proving-broker Proving Broker started
14:37:48 [14:37:48.869]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:26b2daabe6130b99 settled=false
{"provingJobId":"1:BASE_PARITY:26b2daabe6130b99","status":"pending"}
14:37:48 [14:37:48.870]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:81c3617bc7e11b5c settled=false
{"provingJobId":"1:BASE_PARITY:81c3617bc7e11b5c","status":"pending"}
14:37:48 [14:37:48.871]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:26b2daabe6130b99 type=BASE_PARITY startedAt=1749825467817 older agent has taken job
{"provingJobId":"1:BASE_PARITY:26b2daabe6130b99"}
14:37:48 [14:37:48.871]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:26b2daabe6130b99 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:26b2daabe6130b99"}
14:37:48 [14:37:48.873]
INFO:
prover-client:proving-broker Proving Broker started
14:37:48 [14:37:48.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6947c28fbb04c442 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6947c28fbb04c442"}
14:37:48 [14:37:48.874]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c53870950f555132 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c53870950f555132"}
14:37:49 [14:37:49.874]
INFO:
prover-client:proving-broker Proving Broker started
14:37:49 [14:37:49.875]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6947c28fbb04c442 settled=false
{"provingJobId":"1:BASE_PARITY:6947c28fbb04c442","status":"pending"}
14:37:49 [14:37:49.875]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c53870950f555132 settled=false
{"provingJobId":"1:BASE_PARITY:c53870950f555132","status":"pending"}
14:37:49 [14:37:49.876]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6947c28fbb04c442 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:6947c28fbb04c442"}
14:37:49 [14:37:49.877]
INFO:
prover-client:proving-broker Proving Broker started
14:37:49 [14:37:49.878]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08b4d9ef89da0651 epochNumber=1
{"provingJobId":"1:BASE_PARITY:08b4d9ef89da0651"}
14:37:49 [14:37:49.878]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05a173b470bc72bd epochNumber=2
{"provingJobId":"1:BASE_PARITY:05a173b470bc72bd"}
14:37:50 [14:37:50.879]
INFO:
prover-client:proving-broker Proving Broker started
14:37:50 [14:37:50.880]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:08b4d9ef89da0651 settled=false
{"provingJobId":"1:BASE_PARITY:08b4d9ef89da0651","status":"pending"}
14:37:50 [14:37:50.880]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:05a173b470bc72bd settled=false
{"provingJobId":"1:BASE_PARITY:05a173b470bc72bd","status":"pending"}
14:37:50 [14:37:50.881]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:08b4d9ef89da0651 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:08b4d9ef89da0651"}
14:37:50 [14:37:50.881]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:08b4d9ef89da0651 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:08b4d9ef89da0651"}
14:37:50 [14:37:50.882]
INFO:
prover-client:proving-broker Proving Broker started
14:37:50 [14:37:50.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb696762e207a22c epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb696762e207a22c"}
14:37:50 [14:37:50.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9511e9dc24c484c epochNumber=2
{"provingJobId":"1:BASE_PARITY:e9511e9dc24c484c"}
14:37:50 [14:37:50.883]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bb696762e207a22c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bb696762e207a22c"}
14:37:50 [14:37:50.884]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e9511e9dc24c484c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e9511e9dc24c484c"}
14:37:50 [14:37:50.885]
INFO:
prover-client:proving-broker Proving Broker started
14:37:50 [14:37:50.885]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d8e164686442036b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d8e164686442036b"}
14:37:50 [14:37:50.886]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9cf0cd098cd433c6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9cf0cd098cd433c6"}
14:37:50 [14:37:50.886]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d8e164686442036b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d8e164686442036b"}
14:37:50 [14:37:50.886]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d8e164686442036b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d8e164686442036b"}
14:37:50 [14:37:50.886]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9cf0cd098cd433c6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9cf0cd098cd433c6"}
14:37:50 [14:37:50.887]
INFO:
prover-client:proving-broker Proving Broker started
14:37:50 [14:37:50.888]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:5888596c82bcd486 err=test error
{"provingJoId":"1:BASE_PARITY:5888596c82bcd486"}
14:37:50 [14:37:50.889]
INFO:
prover-client:proving-broker Proving Broker started
14:37:50 [14:37:50.889]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e394a9e2ae4373d1 not found
{"provingJobId":"1:BASE_PARITY:e394a9e2ae4373d1"}
14:37:50 [14:37:50.890]
INFO:
prover-client:proving-broker Proving Broker started
14:37:50 [14:37:50.891]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b291a912ec86265d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b291a912ec86265d"}
14:37:50 [14:37:50.892]
INFO:
prover-client:proving-broker Proving Broker started
14:37:50 [14:37:50.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5271d79a8acf68e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5271d79a8acf68e5"}
14:37:50 [14:37:50.993]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5271d79a8acf68e5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5271d79a8acf68e5"}
14:37:51 [14:37:51.001]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.001]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f72de69e7fb5ede4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f72de69e7fb5ede4"}
14:37:51 [14:37:51.102]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f72de69e7fb5ede4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f72de69e7fb5ede4"}
14:37:51 [14:37:51.108]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c2c872bbc69e985 epochNumber=10
{"provingJobId":"1:BASE_PARITY:8c2c872bbc69e985"}
14:37:51 [14:37:51.128]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:51 [14:37:51.235]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e09af53fa8ecf3d8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e09af53fa8ecf3d8"}
14:37:51 [14:37:51.387]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e09af53fa8ecf3d8 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e09af53fa8ecf3d8"}
14:37:51 [14:37:51.438]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34045f42799afa65 epochNumber=1
{"provingJobId":"1:BASE_PARITY:34045f42799afa65"}
14:37:51 [14:37:51.439]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:34045f42799afa65 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:34045f42799afa65"}
14:37:51 [14:37:51.440]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.441]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b683d17679294510 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b683d17679294510"}
14:37:51 [14:37:51.441]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b683d17679294510 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b683d17679294510"}
14:37:51 [14:37:51.442]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b683d17679294510 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:b683d17679294510"}
14:37:51 [14:37:51.444]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.445]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c6f6c98535ad0f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c6f6c98535ad0f0"}
14:37:51 [14:37:51.445]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0c6f6c98535ad0f0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0c6f6c98535ad0f0"}
14:37:51 [14:37:51.446]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90d726199b89f5bb epochNumber=1
{"provingJobId":"1:BASE_PARITY:90d726199b89f5bb"}
14:37:51 [14:37:51.448]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:90d726199b89f5bb type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:90d726199b89f5bb"}
14:37:51 [14:37:51.448]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a6cd4ac6b5a3513 epochNumber=3
{"provingJobId":"1:BASE_PARITY:3a6cd4ac6b5a3513"}
14:37:51 [14:37:51.472]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:51 [14:37:51.473]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:90d726199b89f5bb err=test error
{"provingJoId":"1:BASE_PARITY:90d726199b89f5bb"}
14:37:51 [14:37:51.475]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.476]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:69e8d18796f4c820 settled=false
{"provingJobId":"1:BASE_PARITY:69e8d18796f4c820","status":"pending"}
14:37:51 [14:37:51.476]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2579768ae9a7b504 settled=false
{"provingJobId":"1:BASE_PARITY:2579768ae9a7b504","status":"pending"}
14:37:51 [14:37:51.478]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.479]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d1dcab8a26743eb2 settled=true
{"provingJobId":"1:BASE_PARITY:d1dcab8a26743eb2","status":"fulfilled"}
14:37:51 [14:37:51.479]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:30bdb7c9be847348 settled=true
{"provingJobId":"2:BASE_PARITY:30bdb7c9be847348","status":"fulfilled"}
14:37:51 [14:37:51.480]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.481]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2c858b0a15155193 settled=true
{"provingJobId":"1:BASE_PARITY:2c858b0a15155193","status":"fulfilled"}
14:37:51 [14:37:51.481]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:39a765c95e428e6b settled=false
{"provingJobId":"1:BASE_PARITY:39a765c95e428e6b","status":"pending"}
14:37:51 [14:37:51.482]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c07c8360a97f5c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c07c8360a97f5c4"}
14:37:51 [14:37:51.487]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.488]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11c31d3aa9e7b418 epochNumber=1
{"provingJobId":"1:BASE_PARITY:11c31d3aa9e7b418"}
14:37:51 [14:37:51.489]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:11c31d3aa9e7b418: Error: db error: Error: db error
14:37:51 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:11c31d3aa9e7b418"}
14:37:51 [14:37:51.490]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.490]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d64aea5e9bbce8b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d64aea5e9bbce8b"}
14:37:51 [14:37:51.491]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4d64aea5e9bbce8b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4d64aea5e9bbce8b"}
14:37:51 [14:37:51.491]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4d64aea5e9bbce8b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4d64aea5e9bbce8b"}
14:37:51 [14:37:51.492]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.493]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eddb5ea039b7bd4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:eddb5ea039b7bd4b"}
14:37:51 [14:37:51.493]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eddb5ea039b7bd4b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:eddb5ea039b7bd4b"}
14:37:51 [14:37:51.493]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:eddb5ea039b7bd4b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:eddb5ea039b7bd4b"}
14:37:51 [14:37:51.494]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:eddb5ea039b7bd4b: Error: db error
14:37:51 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:eddb5ea039b7bd4b"}
14:37:51 [14:37:51.495]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.496]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c657158f759b5995 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c657158f759b5995"}
14:37:51 [14:37:51.496]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c657158f759b5995 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c657158f759b5995"}
14:37:51 [14:37:51.496]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c657158f759b5995 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c657158f759b5995"}
14:37:51 [14:37:51.498]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.498]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df347fdce8413741 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df347fdce8413741"}
14:37:51 [14:37:51.498]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:df347fdce8413741 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:df347fdce8413741"}
14:37:51 [14:37:51.498]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:df347fdce8413741 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:df347fdce8413741"}
14:37:51 [14:37:51.499]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:df347fdce8413741 jobErr=test error: Error: db error
14:37:51 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:df347fdce8413741"}
14:37:51 [14:37:51.500]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.501]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1657e07bf162a961 not found
{"provingJobId":"1:BASE_PARITY:1657e07bf162a961"}
14:37:51 [14:37:51.502]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.503]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:09eabe68e8df2fcd err=test error
{"provingJoId":"1:BASE_PARITY:09eabe68e8df2fcd"}
14:37:51 [14:37:51.504]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.514]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e65e54cabaf9c7f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e65e54cabaf9c7f8"}
14:37:51 [14:37:51.514]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e65e54cabaf9c7f8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e65e54cabaf9c7f8"}
14:37:51 [14:37:51.515]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e65e54cabaf9c7f8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e65e54cabaf9c7f8"}
14:37:51 [14:37:51.525]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:c88b2326a05f7f2b epochNumber=2
{"provingJobId":"2:BASE_PARITY:c88b2326a05f7f2b"}
14:37:51 [14:37:51.525]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:c88b2326a05f7f2b type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:c88b2326a05f7f2b"}
14:37:51 [14:37:51.526]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c88b2326a05f7f2b type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:c88b2326a05f7f2b"}
14:37:51 [14:37:51.587]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:837214ee862dab98 epochNumber=3
{"provingJobId":"3:BASE_PARITY:837214ee862dab98"}
14:37:51 [14:37:51.605]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:51 [14:37:51.649]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:478cc17f74c13b7a epochNumber=4
{"provingJobId":"4:BASE_PARITY:478cc17f74c13b7a"}
14:37:51 [14:37:51.656]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:51 [14:37:51.711]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:ebab786f34e7b1cf epochNumber=5
{"provingJobId":"5:BASE_PARITY:ebab786f34e7b1cf"}
14:37:51 [14:37:51.733]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:51 [14:37:51.766]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.767]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:750ddfaa5a81d969 epochNumber=42
{"provingJobId":"1:BASE_PARITY:750ddfaa5a81d969"}
14:37:51 [14:37:51.769]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testshrQ0v/42 with map size 1073741824
14:37:51 [14:37:51.773]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:51 [14:37:51.800]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:2c36880580d40382 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c36880580d40382"}
14:37:51 [14:37:51.805]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7aac4690b106e427 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7aac4690b106e427"}
14:37:51 [14:37:51.806]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFq8YNP/1 with map size 1073741824
14:37:51 [14:37:51.808]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:51 [14:37:51.827]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50bf6598777f07b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:50bf6598777f07b1"}
14:37:51 [14:37:51.835]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.836]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17814e7bcfca4730 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17814e7bcfca4730"}
14:37:51 [14:37:51.839]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlOEzWV/1 with map size 1073741824
14:37:51 [14:37:51.841]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:51 [14:37:51.858]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:17814e7bcfca4730 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:17814e7bcfca4730"}
14:37:51 [14:37:51.862]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b34770c59035aca epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b34770c59035aca"}
14:37:51 [14:37:51.863]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrrGFFA/1 with map size 1073741824
14:37:51 [14:37:51.864]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:51 [14:37:51.881]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8b34770c59035aca epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8b34770c59035aca"}
14:37:51 [14:37:51.882]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8b34770c59035aca type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8b34770c59035aca"}
14:37:51 [14:37:51.888]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8b34770c59035aca epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8b34770c59035aca"}
14:37:51 [14:37:51.891]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e1f1ff1c152ebaf epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e1f1ff1c152ebaf"}
14:37:51 [14:37:51.893]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDPgBlX/1 with map size 1073741824
14:37:51 [14:37:51.895]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:51 [14:37:51.918]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.921]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.921]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa1bf33547a6a6df epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa1bf33547a6a6df"}
14:37:51 [14:37:51.922]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLFAwbY/1 with map size 1073741824
14:37:51 [14:37:51.924]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:51 [14:37:51.949]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:fa1bf33547a6a6df
{"provingJobId":"1:BASE_PARITY:fa1bf33547a6a6df"}
14:37:51 [14:37:51.949]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fa1bf33547a6a6df type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fa1bf33547a6a6df"}
14:37:51 [14:37:51.950]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fa1bf33547a6a6df type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:fa1bf33547a6a6df"}
14:37:51 [14:37:51.967]
INFO:
prover-client:proving-broker Proving Broker started
14:37:51 [14:37:51.970]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a85a00d90f45114 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a85a00d90f45114"}
14:37:51 [14:37:51.971]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0aaZNw/1 with map size 1073741824
14:37:51 [14:37:51.974]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:51 [14:37:51.995]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7a85a00d90f45114
{"provingJobId":"1:BASE_PARITY:7a85a00d90f45114"}
14:37:51 [14:37:51.996]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7a85a00d90f45114 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7a85a00d90f45114"}
14:37:52 [14:37:52.010]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa8a88691cec75bb epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa8a88691cec75bb"}
14:37:52 [14:37:52.013]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJMcYIT/1 with map size 1073741824
14:37:52 [14:37:52.016]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.034]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fa8a88691cec75bb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fa8a88691cec75bb"}
14:37:52 [14:37:52.035]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fa8a88691cec75bb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fa8a88691cec75bb"}
14:37:52 [14:37:52.049]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.050]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1eb5c806302eb05e epochNumber=1
{"provingJobId":"1:BASE_PARITY:1eb5c806302eb05e"}
14:37:52 [14:37:52.055]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpJ4Jr6/1 with map size 1073741824
14:37:52 [14:37:52.061]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.081]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1eb5c806302eb05e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1eb5c806302eb05e"}
14:37:52 [14:37:52.082]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1eb5c806302eb05e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1eb5c806302eb05e"}
14:37:52 [14:37:52.095]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.097]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be042d9e89e57fc1 epochNumber=0
{"provingJobId":"1:BASE_PARITY:be042d9e89e57fc1"}
14:37:52 [14:37:52.098]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testsElMM7/0 with map size 1073741824
14:37:52 [14:37:52.101]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.124]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:be042d9e89e57fc1 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:be042d9e89e57fc1"}
14:37:52 [14:37:52.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83dbf177c4cdd7a8 epochNumber=0
{"provingJobId":"1:BASE_PARITY:83dbf177c4cdd7a8"}
14:37:52 [14:37:52.127]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:be042d9e89e57fc1 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:be042d9e89e57fc1"}
14:37:52 [14:37:52.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70d416c29811b320 epochNumber=0
{"provingJobId":"1:BASE_PARITY:70d416c29811b320"}
14:37:52 [14:37:52.146]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.152]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.154]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cbbdface71a856ca epochNumber=2
{"provingJobId":"1:BASE_PARITY:cbbdface71a856ca"}
14:37:52 [14:37:52.155]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6DJ5T4/2 with map size 1073741824
14:37:52 [14:37:52.164]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.191]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca6c69146e3228ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca6c69146e3228ac"}
14:37:52 [14:37:52.192]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6DJ5T4/1 with map size 1073741824
14:37:52 [14:37:52.197]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.225]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.226]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d298d950bf126426 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d298d950bf126426"}
14:37:52 [14:37:52.227]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrmRc4Y/1 with map size 1073741824
14:37:52 [14:37:52.230]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.259]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a65a0ad4ac6d8b0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a65a0ad4ac6d8b0f"}
14:37:52 [14:37:52.269]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvtEi8F/1 with map size 1073741824
14:37:52 [14:37:52.271]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.297]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0b8aeaf479c6456 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0b8aeaf479c6456"}
14:37:52 [14:37:52.306]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5fc4e7b74c14b9cb epochNumber=2
{"provingJobId":"1:BASE_PARITY:5fc4e7b74c14b9cb"}
14:37:52 [14:37:52.308]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvtEi8F/2 with map size 1073741824
14:37:52 [14:37:52.313]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.338]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d06605c03f0ca93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d06605c03f0ca93"}
14:37:52 [14:37:52.359]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.360]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a4c6d34fababe84b epochNumber=1
{"provingJobId":"1:BASE_PARITY:a4c6d34fababe84b"}
14:37:52 [14:37:52.363]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWatZ26/1 with map size 1073741824
14:37:52 [14:37:52.367]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.391]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78f26daadf6731cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:78f26daadf6731cd"}
14:37:52 [14:37:52.403]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2646eb337cdd8af2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2646eb337cdd8af2"}
14:37:52 [14:37:52.406]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWatZ26/2 with map size 1073741824
14:37:52 [14:37:52.416]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22c18b11dc8e33ef epochNumber=1
{"provingJobId":"1:BASE_PARITY:22c18b11dc8e33ef"}
14:37:52 [14:37:52.453]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.454]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a54a7eda859cb73a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a54a7eda859cb73a"}
14:37:52 [14:37:52.456]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH1xmvb/1 with map size 1073741824
14:37:52 [14:37:52.462]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.490]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5c158c6d469043b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5c158c6d469043b"}
14:37:52 [14:37:52.499]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:469f695adbef172b epochNumber=2
{"provingJobId":"1:BASE_PARITY:469f695adbef172b"}
14:37:52 [14:37:52.500]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testH1xmvb/2 with map size 1073741824
14:37:52 [14:37:52.507]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.528]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fca0abd093ae8de8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fca0abd093ae8de8"}
14:37:52 [14:37:52.543]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a301f550c9125ab4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a301f550c9125ab4"}
14:37:52 [14:37:52.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3kinFT/1 with map size 1073741824
14:37:52 [14:37:52.553]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.577]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1d1b8d2d258aec1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1d1b8d2d258aec1"}
14:37:52 [14:37:52.588]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a301f550c9125ab4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a301f550c9125ab4"}
14:37:52 [14:37:52.601]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.602]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ddf6708a3a1b46d epochNumber=1
{"provingJobId":"1:BASE_PARITY:0ddf6708a3a1b46d"}
14:37:52 [14:37:52.604]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testixZPLM/1 with map size 1073741824
14:37:52 [14:37:52.608]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.632]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c469856ee7dc1eb5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c469856ee7dc1eb5"}
14:37:52 [14:37:52.640]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0ddf6708a3a1b46d type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:0ddf6708a3a1b46d"}
14:37:52 [14:37:52.655]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.656]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94f3128588db83ae epochNumber=1
{"provingJobId":"1:BASE_PARITY:94f3128588db83ae"}
14:37:52 [14:37:52.657]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNFl6Qo/1 with map size 1073741824
14:37:52 [14:37:52.658]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.684]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0e081bb2ad30ebb epochNumber=1
{"provingJobId":"1:BASE_PARITY:f0e081bb2ad30ebb"}
14:37:52 [14:37:52.692]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:94f3128588db83ae type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:94f3128588db83ae"}
14:37:52 [14:37:52.698]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.699]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a078e9e7d48f66e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a078e9e7d48f66e9"}
14:37:52 [14:37:52.703]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testceUSW5/1 with map size 1073741824
14:37:52 [14:37:52.707]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.734]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a078e9e7d48f66e9
{"provingJobId":"1:BASE_PARITY:a078e9e7d48f66e9"}
14:37:52 [14:37:52.735]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a078e9e7d48f66e9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a078e9e7d48f66e9"}
14:37:52 [14:37:52.743]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71cc579c9fd8ca53 epochNumber=1
{"provingJobId":"1:BASE_PARITY:71cc579c9fd8ca53"}
14:37:52 [14:37:52.750]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a078e9e7d48f66e9 has already been completed
{"provingJobId":"1:BASE_PARITY:a078e9e7d48f66e9"}
14:37:52 [14:37:52.754]
INFO:
prover-client:proving-broker Proving Broker started
14:37:52 [14:37:52.755]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c1a353c3bdf8ed2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c1a353c3bdf8ed2"}
14:37:52 [14:37:52.756]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkPucSB/1 with map size 1073741824
14:37:52 [14:37:52.757]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:52 [14:37:52.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9678e818837b5243 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9678e818837b5243"}
14:37:52 [14:37:52.783]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkPucSB/2 with map size 1073741824
14:37:52 [14:37:52.785]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:53 [14:37:53.858]
INFO:
prover-client:proving-broker Proving Broker started
14:37:53 [14:37:53.868]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9c1a353c3bdf8ed2 settled=false
{"provingJobId":"1:BASE_PARITY:9c1a353c3bdf8ed2","status":"pending"}
14:37:53 [14:37:53.870]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9678e818837b5243 settled=false
{"provingJobId":"1:BASE_PARITY:9678e818837b5243","status":"pending"}
14:37:53 [14:37:53.871]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9c1a353c3bdf8ed2 type=BASE_PARITY startedAt=1749825472807 older agent has taken job
{"provingJobId":"1:BASE_PARITY:9c1a353c3bdf8ed2"}
14:37:53 [14:37:53.872]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9c1a353c3bdf8ed2 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:9c1a353c3bdf8ed2"}
14:37:53 [14:37:53.877]
INFO:
prover-client:proving-broker Proving Broker started
14:37:53 [14:37:53.880]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c1d151b213989de epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c1d151b213989de"}
14:37:53 [14:37:53.881]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpwhMEX/1 with map size 1073741824
14:37:53 [14:37:53.884]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:53 [14:37:53.907]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1c0ed70f881da6a epochNumber=2
{"provingJobId":"1:BASE_PARITY:d1c0ed70f881da6a"}
14:37:53 [14:37:53.908]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpwhMEX/2 with map size 1073741824
14:37:53 [14:37:53.911]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:54 [14:37:54.931]
INFO:
prover-client:proving-broker Proving Broker started
14:37:54 [14:37:54.933]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8c1d151b213989de settled=false
{"provingJobId":"1:BASE_PARITY:8c1d151b213989de","status":"pending"}
14:37:54 [14:37:54.935]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d1c0ed70f881da6a settled=false
{"provingJobId":"1:BASE_PARITY:d1c0ed70f881da6a","status":"pending"}
14:37:54 [14:37:54.936]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8c1d151b213989de type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:8c1d151b213989de"}
14:37:54 [14:37:54.944]
INFO:
prover-client:proving-broker Proving Broker started
14:37:54 [14:37:54.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e6264090e12991e epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e6264090e12991e"}
14:37:54 [14:37:54.947]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXejzAR/1 with map size 1073741824
14:37:54 [14:37:54.950]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:54 [14:37:54.969]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92f4aad3d2900bea epochNumber=2
{"provingJobId":"1:BASE_PARITY:92f4aad3d2900bea"}
14:37:54 [14:37:54.970]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testXejzAR/2 with map size 1073741824
14:37:54 [14:37:54.973]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:55 [14:37:55.994]
INFO:
prover-client:proving-broker Proving Broker started
14:37:55 [14:37:55.996]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6e6264090e12991e settled=false
{"provingJobId":"1:BASE_PARITY:6e6264090e12991e","status":"pending"}
14:37:55 [14:37:55.998]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:92f4aad3d2900bea settled=false
{"provingJobId":"1:BASE_PARITY:92f4aad3d2900bea","status":"pending"}
14:37:56 [14:37:56.000]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6e6264090e12991e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6e6264090e12991e"}
14:37:56 [14:37:56.000]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6e6264090e12991e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6e6264090e12991e"}
14:37:56 [14:37:56.011]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28cdccbc5440cc6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:28cdccbc5440cc6a"}
14:37:56 [14:37:56.013]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKP1auB/1 with map size 1073741824
14:37:56 [14:37:56.015]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00682a1da8efd55d epochNumber=2
{"provingJobId":"1:BASE_PARITY:00682a1da8efd55d"}
14:37:56 [14:37:56.037]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKP1auB/2 with map size 1073741824
14:37:56 [14:37:56.039]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.061]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:28cdccbc5440cc6a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:28cdccbc5440cc6a"}
14:37:56 [14:37:56.070]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:00682a1da8efd55d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:00682a1da8efd55d"}
14:37:56 [14:37:56.080]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e26035378114b0b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e26035378114b0b1"}
14:37:56 [14:37:56.083]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzFaeGT/1 with map size 1073741824
14:37:56 [14:37:56.086]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.104]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e015b3c150a7f81b epochNumber=2
{"provingJobId":"1:BASE_PARITY:e015b3c150a7f81b"}
14:37:56 [14:37:56.105]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testzFaeGT/2 with map size 1073741824
14:37:56 [14:37:56.108]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.129]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e26035378114b0b1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e26035378114b0b1"}
14:37:56 [14:37:56.129]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e26035378114b0b1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e26035378114b0b1"}
14:37:56 [14:37:56.136]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e015b3c150a7f81b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e015b3c150a7f81b"}
14:37:56 [14:37:56.156]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.160]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2acd4f595c039543 err=test error
{"provingJoId":"1:BASE_PARITY:2acd4f595c039543"}
14:37:56 [14:37:56.166]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.169]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a31f2fcfa5483829 not found
{"provingJobId":"1:BASE_PARITY:a31f2fcfa5483829"}
14:37:56 [14:37:56.173]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.174]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e9837782e792b4c epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e9837782e792b4c"}
14:37:56 [14:37:56.175]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testys8lI2/1 with map size 1073741824
14:37:56 [14:37:56.178]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.207]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d2933cf6d9e3062 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d2933cf6d9e3062"}
14:37:56 [14:37:56.209]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlUREDb/1 with map size 1073741824
14:37:56 [14:37:56.211]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.337]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9d2933cf6d9e3062 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9d2933cf6d9e3062"}
14:37:56 [14:37:56.341]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.342]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c131604bf5e465e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c131604bf5e465e5"}
14:37:56 [14:37:56.343]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvENmlh/1 with map size 1073741824
14:37:56 [14:37:56.356]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.493]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c131604bf5e465e5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c131604bf5e465e5"}
14:37:56 [14:37:56.495]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d87d39304996649 epochNumber=10
{"provingJobId":"1:BASE_PARITY:6d87d39304996649"}
14:37:56 [14:37:56.496]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testvENmlh/10 with map size 1073741824
14:37:56 [14:37:56.499]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.519]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:56 [14:37:56.520]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:37:56 [14:37:56.654]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.655]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b68f7d4145ef77e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b68f7d4145ef77e1"}
14:37:56 [14:37:56.656]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5am13X/1 with map size 1073741824
14:37:56 [14:37:56.660]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.832]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b68f7d4145ef77e1 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b68f7d4145ef77e1"}
14:37:56 [14:37:56.883]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.884]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70ecceae114c5b25 epochNumber=1
{"provingJobId":"1:BASE_PARITY:70ecceae114c5b25"}
14:37:56 [14:37:56.885]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUvUgsJ/1 with map size 1073741824
14:37:56 [14:37:56.889]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.908]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:70ecceae114c5b25 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:70ecceae114c5b25"}
14:37:56 [14:37:56.911]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.912]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe0e1d7663741751 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe0e1d7663741751"}
14:37:56 [14:37:56.914]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDGOqjS/1 with map size 1073741824
14:37:56 [14:37:56.917]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:56.944]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fe0e1d7663741751 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:fe0e1d7663741751"}
14:37:56 [14:37:56.952]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fe0e1d7663741751 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:fe0e1d7663741751"}
14:37:56 [14:37:56.968]
INFO:
prover-client:proving-broker Proving Broker started
14:37:56 [14:37:56.969]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61d95b5e446f0ada epochNumber=1
{"provingJobId":"1:BASE_PARITY:61d95b5e446f0ada"}
14:37:56 [14:37:56.970]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYBSY0x/1 with map size 1073741824
14:37:56 [14:37:56.976]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:56 [14:37:57.000]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:61d95b5e446f0ada type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:61d95b5e446f0ada"}
14:37:57 [14:37:57.013]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.013]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cdbc2dbb788ddf1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:cdbc2dbb788ddf1f"}
14:37:57 [14:37:57.015]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYtlxUN/1 with map size 1073741824
14:37:57 [14:37:57.021]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.043]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cdbc2dbb788ddf1f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:cdbc2dbb788ddf1f"}
14:37:57 [14:37:57.045]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc50fadcf14c8865 epochNumber=3
{"provingJobId":"1:BASE_PARITY:fc50fadcf14c8865"}
14:37:57 [14:37:57.047]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testYtlxUN/3 with map size 1073741824
14:37:57 [14:37:57.062]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.089]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:57 [14:37:57.090]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:37:57 [14:37:57.113]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:cdbc2dbb788ddf1f err=test error
{"provingJoId":"1:BASE_PARITY:cdbc2dbb788ddf1f"}
14:37:57 [14:37:57.121]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwDKMv7/1 with map size 1073741824
14:37:57 [14:37:57.126]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.155]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwDKMv7/2 with map size 1073741824
14:37:57 [14:37:57.164]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.184]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.186]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b9421f4739571ddf settled=false
{"provingJobId":"1:BASE_PARITY:b9421f4739571ddf","status":"pending"}
14:37:57 [14:37:57.189]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bab3eb99e7e53e47 settled=false
{"provingJobId":"1:BASE_PARITY:bab3eb99e7e53e47","status":"pending"}
14:37:57 [14:37:57.198]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuvxcvJ/1 with map size 1073741824
14:37:57 [14:37:57.203]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.229]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testuvxcvJ/2 with map size 1073741824
14:37:57 [14:37:57.235]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.275]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.279]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bfcd0213d6bb39ae settled=true
{"provingJobId":"1:BASE_PARITY:bfcd0213d6bb39ae","status":"fulfilled"}
14:37:57 [14:37:57.281]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b8b1e2526a367c48 settled=true
{"provingJobId":"2:BASE_PARITY:b8b1e2526a367c48","status":"fulfilled"}
14:37:57 [14:37:57.290]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzvXhbo/1 with map size 1073741824
14:37:57 [14:37:57.293]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.321]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testzvXhbo/2 with map size 1073741824
14:37:57 [14:37:57.328]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.363]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.366]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a771ca121d2aec8b settled=true
{"provingJobId":"1:BASE_PARITY:a771ca121d2aec8b","status":"fulfilled"}
14:37:57 [14:37:57.369]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fbc8e0d0dd747721 settled=false
{"provingJobId":"1:BASE_PARITY:fbc8e0d0dd747721","status":"pending"}
14:37:57 [14:37:57.373]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.373]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f9e9c3b347cbcd26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f9e9c3b347cbcd26"}
14:37:57 [14:37:57.375]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCHVqGM/1 with map size 1073741824
14:37:57 [14:37:57.377]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.403]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.404]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:193c612873ab1e6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:193c612873ab1e6c"}
14:37:57 [14:37:57.404]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:193c612873ab1e6c: Error: db error: Error: db error
14:37:57 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:193c612873ab1e6c"}
14:37:57 [14:37:57.409]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.410]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:066d49739c82fd52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:066d49739c82fd52"}
14:37:57 [14:37:57.411]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTUUu3G/1 with map size 1073741824
14:37:57 [14:37:57.417]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.437]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:066d49739c82fd52 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:066d49739c82fd52"}
14:37:57 [14:37:57.438]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:066d49739c82fd52 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:066d49739c82fd52"}
14:37:57 [14:37:57.457]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.458]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a07dede3846cdf30 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a07dede3846cdf30"}
14:37:57 [14:37:57.464]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test56GUxs/1 with map size 1073741824
14:37:57 [14:37:57.467]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.488]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a07dede3846cdf30 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a07dede3846cdf30"}
14:37:57 [14:37:57.489]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a07dede3846cdf30 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a07dede3846cdf30"}
14:37:57 [14:37:57.490]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:a07dede3846cdf30: Error: db error
14:37:57 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:a07dede3846cdf30"}
14:37:57 [14:37:57.495]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.495]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:753afe0c8d630c89 epochNumber=1
{"provingJobId":"1:BASE_PARITY:753afe0c8d630c89"}
14:37:57 [14:37:57.496]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testP8t7YS/1 with map size 1073741824
14:37:57 [14:37:57.500]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.518]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:753afe0c8d630c89 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:753afe0c8d630c89"}
14:37:57 [14:37:57.519]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:753afe0c8d630c89 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:753afe0c8d630c89"}
14:37:57 [14:37:57.533]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6ed7c16737ee4c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d6ed7c16737ee4c2"}
14:37:57 [14:37:57.535]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNSM2tr/1 with map size 1073741824
14:37:57 [14:37:57.537]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.564]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d6ed7c16737ee4c2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d6ed7c16737ee4c2"}
14:37:57 [14:37:57.565]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d6ed7c16737ee4c2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d6ed7c16737ee4c2"}
14:37:57 [14:37:57.567]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:d6ed7c16737ee4c2 jobErr=test error: Error: db error
14:37:57 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:d6ed7c16737ee4c2"}
14:37:57 [14:37:57.573]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.573]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f6feb37068a886fb not found
{"provingJobId":"1:BASE_PARITY:f6feb37068a886fb"}
14:37:57 [14:37:57.577]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.578]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6f2cd4b9288e0519 err=test error
{"provingJoId":"1:BASE_PARITY:6f2cd4b9288e0519"}
14:37:57 [14:37:57.584]
INFO:
prover-client:proving-broker Proving Broker started
14:37:57 [14:37:57.596]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2209d1f3ed660983 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2209d1f3ed660983"}
14:37:57 [14:37:57.597]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbtQ7lE/1 with map size 1073741824
14:37:57 [14:37:57.599]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.620]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2209d1f3ed660983 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2209d1f3ed660983"}
14:37:57 [14:37:57.621]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2209d1f3ed660983 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2209d1f3ed660983"}
14:37:57 [14:37:57.638]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:c5df8ed5e354486e epochNumber=2
{"provingJobId":"2:BASE_PARITY:c5df8ed5e354486e"}
14:37:57 [14:37:57.641]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testbtQ7lE/2 with map size 1073741824
14:37:57 [14:37:57.653]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.679]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:c5df8ed5e354486e type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:c5df8ed5e354486e"}
14:37:57 [14:37:57.680]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c5df8ed5e354486e type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:c5df8ed5e354486e"}
14:37:57 [14:37:57.749]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:3f5f1b8a7f497ebf epochNumber=3
{"provingJobId":"3:BASE_PARITY:3f5f1b8a7f497ebf"}
14:37:57 [14:37:57.751]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testbtQ7lE/3 with map size 1073741824
14:37:57 [14:37:57.756]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.816]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:57 [14:37:57.817]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:37:57 [14:37:57.854]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:aab70e2767634d51 epochNumber=4
{"provingJobId":"4:BASE_PARITY:aab70e2767634d51"}
14:37:57 [14:37:57.860]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testbtQ7lE/4 with map size 1073741824
14:37:57 [14:37:57.866]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:57 [14:37:57.900]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:57 [14:37:57.901]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
14:37:57 [14:37:57.949]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:4e91d078a9d73324 epochNumber=5
{"provingJobId":"5:BASE_PARITY:4e91d078a9d73324"}
14:37:57 [14:37:57.954]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testbtQ7lE/5 with map size 1073741824
14:37:57 [14:37:57.968]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:37:58 [14:37:58.006]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:37:58 [14:37:58.006]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
14:37:58
PASS src/proving_broker/proving_broker.test.ts (
13.488 s)
14:37:58 ProvingBroker
14:37:58 Producer API
14:37:58
✓ refuses stale jobs (40 ms)
14:37:58
✓ enqueues jobs (4 ms)
14:37:58
✓ ignores duplicate jobs (3 ms)
14:37:58
✓ reports correct status when enqueuing repeat jobs (4 ms)
14:37:58
✓ throws an error in case of duplicate job IDs (28 ms)
14:37:58
✓ returns not-found status for non-existing jobs (2 ms)
14:37:58
✓ cancels jobs in queue (3 ms)
14:37:58
✓ cancels jobs in-progress (10 ms)
14:37:58
✓ returns job result if successful (2 ms)
14:37:58
✓ returns job error if failed (2 ms)
14:37:58
✓ correctly returns job status for concurrent writes (2 ms)
14:37:58
✓ refuses stale jobs (39 ms)
14:37:58
✓ enqueues jobs (31 ms)
14:37:58
✓ ignores duplicate jobs (26 ms)
14:37:58
✓ reports correct status when enqueuing repeat jobs (29 ms)
14:37:58
✓ throws an error in case of duplicate job IDs (26 ms)
14:37:58
✓ returns not-found status for non-existing jobs (3 ms)
14:37:58
✓ cancels jobs in queue (44 ms)
14:37:58
✓ cancels jobs in-progress (42 ms)
14:37:58
✓ returns job result if successful (39 ms)
14:37:58
✓ returns job error if failed (46 ms)
14:37:58
✓ correctly returns job status for concurrent writes (52 ms)
14:37:58 Consumer API
14:37:58
✓ returns undefined if no jobs are available (1 ms)
14:37:58
✓ returns jobs in priority order (1 ms)
14:37:58
✓ returns undefined if no jobs are available for the given allowList (1 ms)
14:37:58
✓ returns a job if it is in the allowList (3 ms)
14:37:58
✓ returns the most important job if it is in the allowList (2 ms)
14:37:58
✓ returns any job if filter is empty (2 ms)
14:37:58
✓ returns a new job when reporting job success (2 ms)
14:37:58
✓ returns a new job when reporting permanent error (2 ms)
14:37:58
✓ returns a new job when reporting retry-able error (2 ms)
14:37:58
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
14:37:58
✓ returns a new job if job is already in progress elsewhere (1056 ms)
14:37:58
✓ avoids sending the same job to a new agent after a restart (1005 ms)
14:37:58
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
14:37:58
✓ tracks job result if in progress (3 ms)
14:37:58
✓ tracks job result even if job is in queue (2 ms)
14:37:58
✓ ignores reported job error if unknown job (1 ms)
14:37:58
✓ ignores job result if unknown job (2 ms)
14:37:58
✓ returns undefined if no jobs are available (3 ms)
14:37:58
✓ returns jobs in priority order (76 ms)
14:37:58
✓ returns undefined if no jobs are available for the given allowList (33 ms)
14:37:58
✓ returns a job if it is in the allowList (95 ms)
14:37:58
✓ returns the most important job if it is in the allowList (96 ms)
14:37:58
✓ returns any job if filter is empty (88 ms)
14:37:58
✓ returns a new job when reporting job success (57 ms)
14:37:58
✓ returns a new job when reporting permanent error (52 ms)
14:37:58
✓ returns a new job when reporting retry-able error (44 ms)
14:37:58
✓ returns a new job when reporting progress if current one is cancelled (57 ms)
14:37:58
✓ returns a new job if job is already in progress elsewhere (1121 ms)
14:37:58
✓ avoids sending the same job to a new agent after a restart (1065 ms)
14:37:58
✓ avoids sending a completed job to a new agent after a restart (1067 ms)
14:37:58
✓ tracks job result if in progress (70 ms)
14:37:58
✓ tracks job result even if job is in queue (65 ms)
14:37:58
✓ ignores reported job error if unknown job (17 ms)
14:37:58
✓ ignores job result if unknown job (8 ms)
14:37:58 Timeouts
14:37:58
✓ tracks in progress jobs (2 ms)
14:37:58
✓ re-enqueues jobs that time out (108 ms)
14:37:58
✓ cancel stale jobs that time out (234 ms)
14:37:58
✓ keeps the jobs in progress while it is alive (202 ms)
14:37:58
✓ tracks in progress jobs (32 ms)
14:37:58
✓ re-enqueues jobs that time out (136 ms)
14:37:58
✓ cancel stale jobs that time out (304 ms)
14:37:58
✓ keeps the jobs in progress while it is alive (237 ms)
14:37:58 Retries
14:37:58
✓ retries jobs (3 ms)
14:37:58
✓ retries up to a maximum number of times (3 ms)
14:37:58
✓ passing retry=false does not retry (2 ms)
14:37:58
✓ does not retry if job is stale (28 ms)
14:37:58
✓ retries jobs (27 ms)
14:37:58
✓ retries up to a maximum number of times (56 ms)
14:37:58
✓ passing retry=false does not retry (43 ms)
14:37:58
✓ does not retry if job is stale (107 ms)
14:37:58 Database management
14:37:58
✓ re-enqueues proof requests on start (3 ms)
14:37:58
✓ restores proof results on start (2 ms)
14:37:58
✓ only re-enqueues unfinished jobs (2 ms)
14:37:58
✓ saves job when enqueued (3 ms)
14:37:58
✓ does not retain job if database fails to save (3 ms)
14:37:58
✓ saves job result (2 ms)
14:37:58
✓ saves result even if database fails to save (3 ms)
14:37:58
✓ saves job error (2 ms)
14:37:58
✓ saves job error even if database fails to save (3 ms)
14:37:58
✓ does not save job result if job is unknown (2 ms)
14:37:58
✓ does not save job error if job is unknown (1 ms)
14:37:58
✓ cleans up old jobs periodically (261 ms)
14:37:58
✓ re-enqueues proof requests on start (77 ms)
14:37:58
✓ restores proof results on start (89 ms)
14:37:58
✓ only re-enqueues unfinished jobs (86 ms)
14:37:58
✓ saves job when enqueued (29 ms)
14:37:58
✓ does not retain job if database fails to save (4 ms)
14:37:58
✓ saves job result (45 ms)
14:37:58
✓ saves result even if database fails to save (40 ms)
14:37:58
✓ saves job error (37 ms)
14:37:58
✓ saves job error even if database fails to save (39 ms)
14:37:58
✓ does not save job result if job is unknown (4 ms)
14:37:58
✓ does not save job error if job is unknown (4 ms)
14:37:58
✓ cleans up old jobs periodically (470 ms)
14:37:58
14:37:58
Test Suites: 1 passed, 1 total
14:37:58
Tests: 96 passed, 96 total
14:37:58
Snapshots: 0 total
14:37:58
Time: 13.58 s
14:37:58
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
14:37:58
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?