Parent Log:
http://ci.aztec-labs.com/9d7da2703e486f3c
Command: 404189e96871acec yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/23ff13c6b16a0c1333bf26aa5c213054c92403c3
Env: REF_NAME=gh-readonly-queue/next/pr-15062-a0944d6ab9d5708daa7ff219a932e7b3784c6637 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 17 16:47:12 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15062_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
16:47:16 [16:47:16.393]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.399]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53f55eacc988ce8c epochNumber=42
{"provingJobId":"1:BASE_PARITY:53f55eacc988ce8c"}
16:47:16 [16:47:16.401]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:66386b41989f00c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66386b41989f00c6"}
16:47:16 [16:47:16.436]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:918c9b88642e7208 epochNumber=1
{"provingJobId":"1:BASE_PARITY:918c9b88642e7208"}
16:47:16 [16:47:16.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d30c5600e87d736b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d30c5600e87d736b"}
16:47:16 [16:47:16.441]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.442]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1a158585d163286 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1a158585d163286"}
16:47:16 [16:47:16.443]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a1a158585d163286 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:a1a158585d163286"}
16:47:16 [16:47:16.445]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.446]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5477f219cdcafc36 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5477f219cdcafc36"}
16:47:16 [16:47:16.448]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5477f219cdcafc36 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5477f219cdcafc36"}
16:47:16 [16:47:16.449]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5477f219cdcafc36 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5477f219cdcafc36"}
16:47:16 [16:47:16.450]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5477f219cdcafc36 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5477f219cdcafc36"}
16:47:16 [16:47:16.451]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.451]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae7baa405a0c9d6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae7baa405a0c9d6c"}
16:47:16 [16:47:16.475]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.477]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64030baab3ba45a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:64030baab3ba45a0"}
16:47:16 [16:47:16.479]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:64030baab3ba45a0
{"provingJobId":"1:BASE_PARITY:64030baab3ba45a0"}
16:47:16 [16:47:16.479]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:64030baab3ba45a0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:64030baab3ba45a0"}
16:47:16 [16:47:16.479]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:64030baab3ba45a0 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:64030baab3ba45a0"}
16:47:16 [16:47:16.481]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b117a939e833095e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b117a939e833095e"}
16:47:16 [16:47:16.483]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b117a939e833095e
{"provingJobId":"1:BASE_PARITY:b117a939e833095e"}
16:47:16 [16:47:16.483]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b117a939e833095e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b117a939e833095e"}
16:47:16 [16:47:16.485]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.486]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d9636c2be99b3fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:1d9636c2be99b3fa"}
16:47:16 [16:47:16.486]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1d9636c2be99b3fa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1d9636c2be99b3fa"}
16:47:16 [16:47:16.486]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1d9636c2be99b3fa type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1d9636c2be99b3fa"}
16:47:16 [16:47:16.488]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.488]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d49cba4f2f64dc4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d49cba4f2f64dc4"}
16:47:16 [16:47:16.489]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8d49cba4f2f64dc4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8d49cba4f2f64dc4"}
16:47:16 [16:47:16.489]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8d49cba4f2f64dc4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8d49cba4f2f64dc4"}
16:47:16 [16:47:16.490]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.490]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e797c13bc30a0b84 epochNumber=0
{"provingJobId":"1:BASE_PARITY:e797c13bc30a0b84"}
16:47:16 [16:47:16.491]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e797c13bc30a0b84 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e797c13bc30a0b84"}
16:47:16 [16:47:16.491]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52d0f45453f0fafc epochNumber=0
{"provingJobId":"1:BASE_PARITY:52d0f45453f0fafc"}
16:47:16 [16:47:16.491]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e797c13bc30a0b84 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e797c13bc30a0b84"}
16:47:16 [16:47:16.491]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b6ec73369085520 epochNumber=0
{"provingJobId":"1:BASE_PARITY:8b6ec73369085520"}
16:47:16 [16:47:16.492]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.493]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.494]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75e702121b211baa epochNumber=2
{"provingJobId":"1:BASE_PARITY:75e702121b211baa"}
16:47:16 [16:47:16.494]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb26efaf11d422a9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb26efaf11d422a9"}
16:47:16 [16:47:16.495]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.495]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b733ae1973c416d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b733ae1973c416d1"}
16:47:16 [16:47:16.505]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8dcbbdef4f5dd04b epochNumber=1
{"provingJobId":"1:BASE_PARITY:8dcbbdef4f5dd04b"}
16:47:16 [16:47:16.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a0a6ff5eae5f861 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a0a6ff5eae5f861"}
16:47:16 [16:47:16.507]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9538902271db298 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b9538902271db298"}
16:47:16 [16:47:16.507]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4bde370a54018341 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4bde370a54018341"}
16:47:16 [16:47:16.508]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.509]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de0772f1b5a1c4fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:de0772f1b5a1c4fb"}
16:47:16 [16:47:16.509]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3d293fa800c0c9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:e3d293fa800c0c9f"}
16:47:16 [16:47:16.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6985a75667b2cf49 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6985a75667b2cf49"}
16:47:16 [16:47:16.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f380419f3e871df epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f380419f3e871df"}
16:47:16 [16:47:16.511]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a0178c6dbea7205 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a0178c6dbea7205"}
16:47:16 [16:47:16.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8f0318a08da2e3e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8f0318a08da2e3e"}
16:47:16 [16:47:16.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84bfdebc299dd2f2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:84bfdebc299dd2f2"}
16:47:16 [16:47:16.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7565be9b2bd212f epochNumber=1
{"provingJobId":"1:BASE_PARITY:d7565be9b2bd212f"}
16:47:16 [16:47:16.514]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.514]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e910f2d0cee4255 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e910f2d0cee4255"}
16:47:16 [16:47:16.515]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ff27fed1fb1f3b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ff27fed1fb1f3b5"}
16:47:16 [16:47:16.515]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7e910f2d0cee4255 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7e910f2d0cee4255"}
16:47:16 [16:47:16.516]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.517]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab20228b9231a08b epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab20228b9231a08b"}
16:47:16 [16:47:16.517]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec9f7dbe7dfcea4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec9f7dbe7dfcea4b"}
16:47:16 [16:47:16.517]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ab20228b9231a08b type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:ab20228b9231a08b"}
16:47:16 [16:47:16.518]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.518]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1effe6d11fd4dfe9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1effe6d11fd4dfe9"}
16:47:16 [16:47:16.519]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c67db868e214050c epochNumber=1
{"provingJobId":"1:BASE_PARITY:c67db868e214050c"}
16:47:16 [16:47:16.519]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1effe6d11fd4dfe9 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:1effe6d11fd4dfe9"}
16:47:16 [16:47:16.520]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.521]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2bf9d6ce8cf4af08 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2bf9d6ce8cf4af08"}
16:47:16 [16:47:16.521]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2bf9d6ce8cf4af08
{"provingJobId":"1:BASE_PARITY:2bf9d6ce8cf4af08"}
16:47:16 [16:47:16.521]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2bf9d6ce8cf4af08 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2bf9d6ce8cf4af08"}
16:47:16 [16:47:16.522]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03a1b729cbd344f6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:03a1b729cbd344f6"}
16:47:16 [16:47:16.522]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2bf9d6ce8cf4af08 has already been completed
{"provingJobId":"1:BASE_PARITY:2bf9d6ce8cf4af08"}
16:47:16 [16:47:16.523]
INFO:
prover-client:proving-broker Proving Broker started
16:47:16 [16:47:16.524]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24f8971a952609df epochNumber=1
{"provingJobId":"1:BASE_PARITY:24f8971a952609df"}
16:47:16 [16:47:16.524]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e090bfe9359704b1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e090bfe9359704b1"}
16:47:17 [16:47:17.577]
INFO:
prover-client:proving-broker Proving Broker started
16:47:17 [16:47:17.578]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:24f8971a952609df settled=false
{"provingJobId":"1:BASE_PARITY:24f8971a952609df","status":"pending"}
16:47:17 [16:47:17.578]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e090bfe9359704b1 settled=false
{"provingJobId":"1:BASE_PARITY:e090bfe9359704b1","status":"pending"}
16:47:17 [16:47:17.579]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:24f8971a952609df type=BASE_PARITY startedAt=1750178836524 older agent has taken job
{"provingJobId":"1:BASE_PARITY:24f8971a952609df"}
16:47:17 [16:47:17.579]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:24f8971a952609df type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:24f8971a952609df"}
16:47:17 [16:47:17.586]
INFO:
prover-client:proving-broker Proving Broker started
16:47:17 [16:47:17.587]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b59f2d0088f466c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b59f2d0088f466c7"}
16:47:17 [16:47:17.587]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5cc8850fd26d361d epochNumber=2
{"provingJobId":"1:BASE_PARITY:5cc8850fd26d361d"}
16:47:18 [16:47:18.588]
INFO:
prover-client:proving-broker Proving Broker started
16:47:18 [16:47:18.590]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b59f2d0088f466c7 settled=false
{"provingJobId":"1:BASE_PARITY:b59f2d0088f466c7","status":"pending"}
16:47:18 [16:47:18.591]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5cc8850fd26d361d settled=false
{"provingJobId":"1:BASE_PARITY:5cc8850fd26d361d","status":"pending"}
16:47:18 [16:47:18.591]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b59f2d0088f466c7 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:b59f2d0088f466c7"}
16:47:18 [16:47:18.593]
INFO:
prover-client:proving-broker Proving Broker started
16:47:18 [16:47:18.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ec771ff1b456077 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3ec771ff1b456077"}
16:47:18 [16:47:18.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:365956d639a6e4bd epochNumber=2
{"provingJobId":"1:BASE_PARITY:365956d639a6e4bd"}
16:47:19 [16:47:19.595]
INFO:
prover-client:proving-broker Proving Broker started
16:47:19 [16:47:19.596]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3ec771ff1b456077 settled=false
{"provingJobId":"1:BASE_PARITY:3ec771ff1b456077","status":"pending"}
16:47:19 [16:47:19.597]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:365956d639a6e4bd settled=false
{"provingJobId":"1:BASE_PARITY:365956d639a6e4bd","status":"pending"}
16:47:19 [16:47:19.599]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3ec771ff1b456077 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3ec771ff1b456077"}
16:47:19 [16:47:19.600]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3ec771ff1b456077 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3ec771ff1b456077"}
16:47:19 [16:47:19.602]
INFO:
prover-client:proving-broker Proving Broker started
16:47:19 [16:47:19.604]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a7dff928adbf7e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a7dff928adbf7e8"}
16:47:19 [16:47:19.604]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:140819f3cc3aa2b8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:140819f3cc3aa2b8"}
16:47:19 [16:47:19.605]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5a7dff928adbf7e8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5a7dff928adbf7e8"}
16:47:19 [16:47:19.606]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:140819f3cc3aa2b8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:140819f3cc3aa2b8"}
16:47:19 [16:47:19.608]
INFO:
prover-client:proving-broker Proving Broker started
16:47:19 [16:47:19.609]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7aa253972a27d1ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:7aa253972a27d1ea"}
16:47:19 [16:47:19.609]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e65f52d9b95b770b epochNumber=2
{"provingJobId":"1:BASE_PARITY:e65f52d9b95b770b"}
16:47:19 [16:47:19.610]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7aa253972a27d1ea type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7aa253972a27d1ea"}
16:47:19 [16:47:19.610]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7aa253972a27d1ea type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7aa253972a27d1ea"}
16:47:19 [16:47:19.610]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e65f52d9b95b770b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e65f52d9b95b770b"}
16:47:19 [16:47:19.612]
INFO:
prover-client:proving-broker Proving Broker started
16:47:19 [16:47:19.612]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b9b901117b3efeea err=test error
{"provingJoId":"1:BASE_PARITY:b9b901117b3efeea"}
16:47:19 [16:47:19.614]
INFO:
prover-client:proving-broker Proving Broker started
16:47:19 [16:47:19.615]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ca219e4d02341c85 not found
{"provingJobId":"1:BASE_PARITY:ca219e4d02341c85"}
16:47:19 [16:47:19.616]
INFO:
prover-client:proving-broker Proving Broker started
16:47:19 [16:47:19.617]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fda140766f86fc30 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fda140766f86fc30"}
16:47:19 [16:47:19.618]
INFO:
prover-client:proving-broker Proving Broker started
16:47:19 [16:47:19.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e61fb3dbc0344c41 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e61fb3dbc0344c41"}
16:47:19 [16:47:19.721]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e61fb3dbc0344c41 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e61fb3dbc0344c41"}
16:47:19 [16:47:19.726]
INFO:
prover-client:proving-broker Proving Broker started
16:47:19 [16:47:19.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ef9a7c318519dbb epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ef9a7c318519dbb"}
16:47:19 [16:47:19.828]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8ef9a7c318519dbb timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8ef9a7c318519dbb"}
16:47:19 [16:47:19.835]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70dbb479ea9b72fe epochNumber=10
{"provingJobId":"1:BASE_PARITY:70dbb479ea9b72fe"}
16:47:19 [16:47:19.854]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:19 [16:47:19.962]
INFO:
prover-client:proving-broker Proving Broker started
16:47:19 [16:47:19.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d60022deaaf57d90 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d60022deaaf57d90"}
16:47:20 [16:47:20.116]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d60022deaaf57d90 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:d60022deaaf57d90"}
16:47:20 [16:47:20.165]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.166]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:562d4587d3899ba1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:562d4587d3899ba1"}
16:47:20 [16:47:20.166]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:562d4587d3899ba1 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:562d4587d3899ba1"}
16:47:20 [16:47:20.167]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0b317f31568405e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0b317f31568405e"}
16:47:20 [16:47:20.168]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b0b317f31568405e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b0b317f31568405e"}
16:47:20 [16:47:20.169]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b0b317f31568405e type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:b0b317f31568405e"}
16:47:20 [16:47:20.170]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.171]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d05f0b23eeaef3f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d05f0b23eeaef3f2"}
16:47:20 [16:47:20.171]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d05f0b23eeaef3f2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d05f0b23eeaef3f2"}
16:47:20 [16:47:20.173]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.174]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6015b696cc0c76b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6015b696cc0c76b4"}
16:47:20 [16:47:20.175]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6015b696cc0c76b4 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6015b696cc0c76b4"}
16:47:20 [16:47:20.176]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fcada98033cb332f epochNumber=3
{"provingJobId":"1:BASE_PARITY:fcada98033cb332f"}
16:47:20 [16:47:20.199]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:20 [16:47:20.201]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6015b696cc0c76b4 err=test error
{"provingJoId":"1:BASE_PARITY:6015b696cc0c76b4"}
16:47:20 [16:47:20.203]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.203]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ecfa1e2f0b57be67 settled=false
{"provingJobId":"1:BASE_PARITY:ecfa1e2f0b57be67","status":"pending"}
16:47:20 [16:47:20.204]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5cb31d90a07bbe9b settled=false
{"provingJobId":"1:BASE_PARITY:5cb31d90a07bbe9b","status":"pending"}
16:47:20 [16:47:20.206]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.207]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:87cff40a20797ac2 settled=true
{"provingJobId":"1:BASE_PARITY:87cff40a20797ac2","status":"fulfilled"}
16:47:20 [16:47:20.207]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:2f83603141421980 settled=true
{"provingJobId":"2:BASE_PARITY:2f83603141421980","status":"fulfilled"}
16:47:20 [16:47:20.208]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.208]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fdeb3d98ad5db323 settled=true
{"provingJobId":"1:BASE_PARITY:fdeb3d98ad5db323","status":"fulfilled"}
16:47:20 [16:47:20.209]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e4cd1cc122f8d77a settled=false
{"provingJobId":"1:BASE_PARITY:e4cd1cc122f8d77a","status":"pending"}
16:47:20 [16:47:20.210]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.211]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e62ebaa1d84c007a epochNumber=1
{"provingJobId":"1:BASE_PARITY:e62ebaa1d84c007a"}
16:47:20 [16:47:20.212]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.213]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:649e66b4d5bde752 epochNumber=1
{"provingJobId":"1:BASE_PARITY:649e66b4d5bde752"}
16:47:20 [16:47:20.214]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:649e66b4d5bde752: Error: db error: Error: db error
16:47:20 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:649e66b4d5bde752"}
16:47:20 [16:47:20.215]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.215]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ea04cdf8ff5200a epochNumber=1
{"provingJobId":"1:BASE_PARITY:3ea04cdf8ff5200a"}
16:47:20 [16:47:20.216]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3ea04cdf8ff5200a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3ea04cdf8ff5200a"}
16:47:20 [16:47:20.216]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3ea04cdf8ff5200a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3ea04cdf8ff5200a"}
16:47:20 [16:47:20.217]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.217]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0eab0f73039e75bd epochNumber=1
{"provingJobId":"1:BASE_PARITY:0eab0f73039e75bd"}
16:47:20 [16:47:20.218]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0eab0f73039e75bd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0eab0f73039e75bd"}
16:47:20 [16:47:20.218]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0eab0f73039e75bd type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0eab0f73039e75bd"}
16:47:20 [16:47:20.219]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:0eab0f73039e75bd: Error: db error
16:47:20 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:0eab0f73039e75bd"}
16:47:20 [16:47:20.220]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.220]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0aca0fc3a3c3903d epochNumber=1
{"provingJobId":"1:BASE_PARITY:0aca0fc3a3c3903d"}
16:47:20 [16:47:20.220]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0aca0fc3a3c3903d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0aca0fc3a3c3903d"}
16:47:20 [16:47:20.221]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0aca0fc3a3c3903d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0aca0fc3a3c3903d"}
16:47:20 [16:47:20.222]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:727f432e4b0619cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:727f432e4b0619cb"}
16:47:20 [16:47:20.222]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:727f432e4b0619cb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:727f432e4b0619cb"}
16:47:20 [16:47:20.223]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:727f432e4b0619cb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:727f432e4b0619cb"}
16:47:20 [16:47:20.223]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:727f432e4b0619cb jobErr=test error: Error: db error
16:47:20 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:727f432e4b0619cb"}
16:47:20 [16:47:20.224]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.225]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:de4e1f3cf3807b7e not found
{"provingJobId":"1:BASE_PARITY:de4e1f3cf3807b7e"}
16:47:20 [16:47:20.229]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.230]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b1b89c52d72778c1 err=test error
{"provingJoId":"1:BASE_PARITY:b1b89c52d72778c1"}
16:47:20 [16:47:20.231]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.242]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f5de603ce4798b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f5de603ce4798b5"}
16:47:20 [16:47:20.243]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6f5de603ce4798b5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6f5de603ce4798b5"}
16:47:20 [16:47:20.243]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6f5de603ce4798b5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6f5de603ce4798b5"}
16:47:20 [16:47:20.254]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:35988385a472cbea epochNumber=2
{"provingJobId":"2:BASE_PARITY:35988385a472cbea"}
16:47:20 [16:47:20.255]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:35988385a472cbea type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:35988385a472cbea"}
16:47:20 [16:47:20.256]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:35988385a472cbea type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:35988385a472cbea"}
16:47:20 [16:47:20.317]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:a504b1fe6b59c6ce epochNumber=3
{"provingJobId":"3:BASE_PARITY:a504b1fe6b59c6ce"}
16:47:20 [16:47:20.332]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:20 [16:47:20.380]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:f4178aa450b86042 epochNumber=4
{"provingJobId":"4:BASE_PARITY:f4178aa450b86042"}
16:47:20 [16:47:20.384]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:20 [16:47:20.443]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:03fe1d85a2866d10 epochNumber=5
{"provingJobId":"5:BASE_PARITY:03fe1d85a2866d10"}
16:47:20 [16:47:20.459]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:20 [16:47:20.498]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.498]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:189a672e99790ffd epochNumber=42
{"provingJobId":"1:BASE_PARITY:189a672e99790ffd"}
16:47:20 [16:47:20.499]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testzhgYXl/42 with map size 1073741824
16:47:20 [16:47:20.504]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.526]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:6610a1be6a4b0665 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6610a1be6a4b0665"}
16:47:20 [16:47:20.533]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbf9ae36fe84d287 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dbf9ae36fe84d287"}
16:47:20 [16:47:20.535]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testA1LxL7/1 with map size 1073741824
16:47:20 [16:47:20.537]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.554]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b3112d13b365ee75 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b3112d13b365ee75"}
16:47:20 [16:47:20.563]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.564]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e301c2648f577e4e epochNumber=1
{"provingJobId":"1:BASE_PARITY:e301c2648f577e4e"}
16:47:20 [16:47:20.565]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4hpNRh/1 with map size 1073741824
16:47:20 [16:47:20.568]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.605]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e301c2648f577e4e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e301c2648f577e4e"}
16:47:20 [16:47:20.608]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af8ae16c9a13c1b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af8ae16c9a13c1b4"}
16:47:20 [16:47:20.609]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7FX5MH/1 with map size 1073741824
16:47:20 [16:47:20.610]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.628]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:af8ae16c9a13c1b4 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:af8ae16c9a13c1b4"}
16:47:20 [16:47:20.628]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:af8ae16c9a13c1b4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:af8ae16c9a13c1b4"}
16:47:20 [16:47:20.634]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:af8ae16c9a13c1b4 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:af8ae16c9a13c1b4"}
16:47:20 [16:47:20.638]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.638]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c5a7bb09acf3bcf epochNumber=1
{"provingJobId":"1:BASE_PARITY:7c5a7bb09acf3bcf"}
16:47:20 [16:47:20.639]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmwNCLK/1 with map size 1073741824
16:47:20 [16:47:20.641]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.663]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.666]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.667]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9df8bf9cae9f6ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9df8bf9cae9f6ea"}
16:47:20 [16:47:20.668]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTyI3Ok/1 with map size 1073741824
16:47:20 [16:47:20.669]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.687]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b9df8bf9cae9f6ea
{"provingJobId":"1:BASE_PARITY:b9df8bf9cae9f6ea"}
16:47:20 [16:47:20.687]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b9df8bf9cae9f6ea type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b9df8bf9cae9f6ea"}
16:47:20 [16:47:20.688]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b9df8bf9cae9f6ea type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b9df8bf9cae9f6ea"}
16:47:20 [16:47:20.699]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88ed1268c687a2a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:88ed1268c687a2a7"}
16:47:20 [16:47:20.702]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwZ4DZU/1 with map size 1073741824
16:47:20 [16:47:20.704]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.722]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:88ed1268c687a2a7
{"provingJobId":"1:BASE_PARITY:88ed1268c687a2a7"}
16:47:20 [16:47:20.723]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:88ed1268c687a2a7 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:88ed1268c687a2a7"}
16:47:20 [16:47:20.732]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.733]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c671cb2f6e33dcf epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c671cb2f6e33dcf"}
16:47:20 [16:47:20.734]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjq2JHB/1 with map size 1073741824
16:47:20 [16:47:20.737]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.754]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6c671cb2f6e33dcf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6c671cb2f6e33dcf"}
16:47:20 [16:47:20.755]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6c671cb2f6e33dcf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6c671cb2f6e33dcf"}
16:47:20 [16:47:20.768]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.768]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a8d6863e1049c9a epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a8d6863e1049c9a"}
16:47:20 [16:47:20.769]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTz3TD2/1 with map size 1073741824
16:47:20 [16:47:20.772]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.803]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4a8d6863e1049c9a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4a8d6863e1049c9a"}
16:47:20 [16:47:20.804]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4a8d6863e1049c9a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4a8d6863e1049c9a"}
16:47:20 [16:47:20.814]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.815]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:488420ec8d96b98c epochNumber=0
{"provingJobId":"1:BASE_PARITY:488420ec8d96b98c"}
16:47:20 [16:47:20.815]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testlVCgER/0 with map size 1073741824
16:47:20 [16:47:20.818]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.837]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:488420ec8d96b98c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:488420ec8d96b98c"}
16:47:20 [16:47:20.837]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d25ab679b469a403 epochNumber=0
{"provingJobId":"1:BASE_PARITY:d25ab679b469a403"}
16:47:20 [16:47:20.838]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:488420ec8d96b98c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:488420ec8d96b98c"}
16:47:20 [16:47:20.838]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01234c7d91c0f564 epochNumber=0
{"provingJobId":"1:BASE_PARITY:01234c7d91c0f564"}
16:47:20 [16:47:20.854]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.858]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.859]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8ce078684c02706 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e8ce078684c02706"}
16:47:20 [16:47:20.861]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOisrbZ/2 with map size 1073741824
16:47:20 [16:47:20.864]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.884]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc6fc0172cfd6b4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc6fc0172cfd6b4b"}
16:47:20 [16:47:20.885]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOisrbZ/1 with map size 1073741824
16:47:20 [16:47:20.895]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.923]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.925]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8883abdc6eb0057d epochNumber=1
{"provingJobId":"1:BASE_PARITY:8883abdc6eb0057d"}
16:47:20 [16:47:20.925]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlffkHJ/1 with map size 1073741824
16:47:20 [16:47:20.928]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.950]
INFO:
prover-client:proving-broker Proving Broker started
16:47:20 [16:47:20.951]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d94cf245af37c29 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d94cf245af37c29"}
16:47:20 [16:47:20.951]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testB4gw0G/1 with map size 1073741824
16:47:20 [16:47:20.953]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:20 [16:47:20.969]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e347a98f715c202a epochNumber=1
{"provingJobId":"1:BASE_PARITY:e347a98f715c202a"}
16:47:20 [16:47:20.976]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e898fbd940bfb925 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e898fbd940bfb925"}
16:47:20 [16:47:20.978]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testB4gw0G/2 with map size 1073741824
16:47:20 [16:47:20.985]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.004]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:998a03d4cac26214 epochNumber=1
{"provingJobId":"1:BASE_PARITY:998a03d4cac26214"}
16:47:21 [16:47:21.014]
INFO:
prover-client:proving-broker Proving Broker started
16:47:21 [16:47:21.015]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3886a620505f97a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3886a620505f97a"}
16:47:21 [16:47:21.016]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRjHHFc/1 with map size 1073741824
16:47:21 [16:47:21.017]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.034]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ca534e935cae232 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ca534e935cae232"}
16:47:21 [16:47:21.040]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:631c5c6780a59ab2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:631c5c6780a59ab2"}
16:47:21 [16:47:21.040]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testRjHHFc/2 with map size 1073741824
16:47:21 [16:47:21.042]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.059]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b0b3466ec644383 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b0b3466ec644383"}
16:47:21 [16:47:21.068]
INFO:
prover-client:proving-broker Proving Broker started
16:47:21 [16:47:21.069]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3f97f03144dc71c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3f97f03144dc71c"}
16:47:21 [16:47:21.071]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYK8wBL/1 with map size 1073741824
16:47:21 [16:47:21.072]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.092]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8af6dd4ebb8cfba9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8af6dd4ebb8cfba9"}
16:47:21 [16:47:21.099]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d17d56f9982c5b6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3d17d56f9982c5b6"}
16:47:21 [16:47:21.100]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYK8wBL/2 with map size 1073741824
16:47:21 [16:47:21.104]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.120]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a035c3b828f44dd9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a035c3b828f44dd9"}
16:47:21 [16:47:21.129]
INFO:
prover-client:proving-broker Proving Broker started
16:47:21 [16:47:21.130]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ca320ba3397d719 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1ca320ba3397d719"}
16:47:21 [16:47:21.130]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testevXFU4/1 with map size 1073741824
16:47:21 [16:47:21.134]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.152]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e166c85528228639 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e166c85528228639"}
16:47:21 [16:47:21.159]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1ca320ba3397d719 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1ca320ba3397d719"}
16:47:21 [16:47:21.170]
INFO:
prover-client:proving-broker Proving Broker started
16:47:21 [16:47:21.171]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0c16a6ee5c58ae5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0c16a6ee5c58ae5"}
16:47:21 [16:47:21.172]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhOqs4C/1 with map size 1073741824
16:47:21 [16:47:21.174]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.190]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66d5ff108830d72b epochNumber=1
{"provingJobId":"1:BASE_PARITY:66d5ff108830d72b"}
16:47:21 [16:47:21.196]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b0c16a6ee5c58ae5 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:b0c16a6ee5c58ae5"}
16:47:21 [16:47:21.205]
INFO:
prover-client:proving-broker Proving Broker started
16:47:21 [16:47:21.206]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8798683f28e2b2ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:8798683f28e2b2ac"}
16:47:21 [16:47:21.209]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLBJoKP/1 with map size 1073741824
16:47:21 [16:47:21.211]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d8278a6cccb84dd8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d8278a6cccb84dd8"}
16:47:21 [16:47:21.238]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8798683f28e2b2ac type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:8798683f28e2b2ac"}
16:47:21 [16:47:21.244]
INFO:
prover-client:proving-broker Proving Broker started
16:47:21 [16:47:21.244]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0258e12acfd5b6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0258e12acfd5b6a"}
16:47:21 [16:47:21.245]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSz9zsh/1 with map size 1073741824
16:47:21 [16:47:21.247]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.263]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e0258e12acfd5b6a
{"provingJobId":"1:BASE_PARITY:e0258e12acfd5b6a"}
16:47:21 [16:47:21.264]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e0258e12acfd5b6a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:e0258e12acfd5b6a"}
16:47:21 [16:47:21.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b21d45f814507665 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b21d45f814507665"}
16:47:21 [16:47:21.274]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e0258e12acfd5b6a has already been completed
{"provingJobId":"1:BASE_PARITY:e0258e12acfd5b6a"}
16:47:21 [16:47:21.278]
INFO:
prover-client:proving-broker Proving Broker started
16:47:21 [16:47:21.278]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d5241dfa2c8a1fd1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d5241dfa2c8a1fd1"}
16:47:21 [16:47:21.279]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQvDPAV/1 with map size 1073741824
16:47:21 [16:47:21.282]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:21 [16:47:21.298]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7219c90d3f2392c2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7219c90d3f2392c2"}
16:47:21 [16:47:21.299]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQvDPAV/2 with map size 1073741824
16:47:21 [16:47:21.301]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:22 [16:47:22.370]
INFO:
prover-client:proving-broker Proving Broker started
16:47:22 [16:47:22.376]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d5241dfa2c8a1fd1 settled=false
{"provingJobId":"1:BASE_PARITY:d5241dfa2c8a1fd1","status":"pending"}
16:47:22 [16:47:22.383]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7219c90d3f2392c2 settled=false
{"provingJobId":"1:BASE_PARITY:7219c90d3f2392c2","status":"pending"}
16:47:22 [16:47:22.385]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d5241dfa2c8a1fd1 type=BASE_PARITY startedAt=1750178841317 older agent has taken job
{"provingJobId":"1:BASE_PARITY:d5241dfa2c8a1fd1"}
16:47:22 [16:47:22.385]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d5241dfa2c8a1fd1 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:d5241dfa2c8a1fd1"}
16:47:22 [16:47:22.391]
INFO:
prover-client:proving-broker Proving Broker started
16:47:22 [16:47:22.393]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:462f7c1ddaf1bdbf epochNumber=1
{"provingJobId":"1:BASE_PARITY:462f7c1ddaf1bdbf"}
16:47:22 [16:47:22.394]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAcwBMA/1 with map size 1073741824
16:47:22 [16:47:22.398]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:22 [16:47:22.421]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ea20c1bb24a0533 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5ea20c1bb24a0533"}
16:47:22 [16:47:22.422]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAcwBMA/2 with map size 1073741824
16:47:22 [16:47:22.424]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:23 [16:47:23.443]
INFO:
prover-client:proving-broker Proving Broker started
16:47:23 [16:47:23.449]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:462f7c1ddaf1bdbf settled=false
{"provingJobId":"1:BASE_PARITY:462f7c1ddaf1bdbf","status":"pending"}
16:47:23 [16:47:23.451]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5ea20c1bb24a0533 settled=false
{"provingJobId":"1:BASE_PARITY:5ea20c1bb24a0533","status":"pending"}
16:47:23 [16:47:23.452]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:462f7c1ddaf1bdbf type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:462f7c1ddaf1bdbf"}
16:47:23 [16:47:23.457]
INFO:
prover-client:proving-broker Proving Broker started
16:47:23 [16:47:23.457]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff97a4f8988e97db epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff97a4f8988e97db"}
16:47:23 [16:47:23.458]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdBCTAO/1 with map size 1073741824
16:47:23 [16:47:23.460]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:23 [16:47:23.479]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1e965cbcca9cfeb epochNumber=2
{"provingJobId":"1:BASE_PARITY:e1e965cbcca9cfeb"}
16:47:23 [16:47:23.480]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testdBCTAO/2 with map size 1073741824
16:47:23 [16:47:23.481]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:24 [16:47:24.502]
INFO:
prover-client:proving-broker Proving Broker started
16:47:24 [16:47:24.504]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ff97a4f8988e97db settled=false
{"provingJobId":"1:BASE_PARITY:ff97a4f8988e97db","status":"pending"}
16:47:24 [16:47:24.506]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e1e965cbcca9cfeb settled=false
{"provingJobId":"1:BASE_PARITY:e1e965cbcca9cfeb","status":"pending"}
16:47:24 [16:47:24.508]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ff97a4f8988e97db type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ff97a4f8988e97db"}
16:47:24 [16:47:24.508]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ff97a4f8988e97db type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ff97a4f8988e97db"}
16:47:24 [16:47:24.520]
INFO:
prover-client:proving-broker Proving Broker started
16:47:24 [16:47:24.521]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df0f24a803203821 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df0f24a803203821"}
16:47:24 [16:47:24.522]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNiWDgC/1 with map size 1073741824
16:47:24 [16:47:24.524]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:24 [16:47:24.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c379c725a675cb17 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c379c725a675cb17"}
16:47:24 [16:47:24.542]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNiWDgC/2 with map size 1073741824
16:47:24 [16:47:24.545]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:24 [16:47:24.562]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:df0f24a803203821 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:df0f24a803203821"}
16:47:24 [16:47:24.568]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c379c725a675cb17 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c379c725a675cb17"}
16:47:24 [16:47:24.578]
INFO:
prover-client:proving-broker Proving Broker started
16:47:24 [16:47:24.579]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c2a7029cdb89c09 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c2a7029cdb89c09"}
16:47:24 [16:47:24.581]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrWqZxl/1 with map size 1073741824
16:47:24 [16:47:24.582]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:24 [16:47:24.604]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a0225efea5ba5ea7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a0225efea5ba5ea7"}
16:47:24 [16:47:24.605]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrWqZxl/2 with map size 1073741824
16:47:24 [16:47:24.607]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:24 [16:47:24.627]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0c2a7029cdb89c09 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0c2a7029cdb89c09"}
16:47:24 [16:47:24.627]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0c2a7029cdb89c09 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0c2a7029cdb89c09"}
16:47:24 [16:47:24.636]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a0225efea5ba5ea7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a0225efea5ba5ea7"}
16:47:24 [16:47:24.645]
INFO:
prover-client:proving-broker Proving Broker started
16:47:24 [16:47:24.646]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:fad5d25290f32bd1 err=test error
{"provingJoId":"1:BASE_PARITY:fad5d25290f32bd1"}
16:47:24 [16:47:24.650]
INFO:
prover-client:proving-broker Proving Broker started
16:47:24 [16:47:24.651]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5e4af7cf1a1027f9 not found
{"provingJobId":"1:BASE_PARITY:5e4af7cf1a1027f9"}
16:47:24 [16:47:24.654]
INFO:
prover-client:proving-broker Proving Broker started
16:47:24 [16:47:24.654]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9efe75af07de4d0c epochNumber=1
{"provingJobId":"1:BASE_PARITY:9efe75af07de4d0c"}
16:47:24 [16:47:24.655]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqToZ6A/1 with map size 1073741824
16:47:24 [16:47:24.656]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:24 [16:47:24.677]
INFO:
prover-client:proving-broker Proving Broker started
16:47:24 [16:47:24.677]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:646cf03a62abf305 epochNumber=1
{"provingJobId":"1:BASE_PARITY:646cf03a62abf305"}
16:47:24 [16:47:24.678]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5XfA0N/1 with map size 1073741824
16:47:24 [16:47:24.680]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:24 [16:47:24.804]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:646cf03a62abf305 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:646cf03a62abf305"}
16:47:24 [16:47:24.808]
INFO:
prover-client:proving-broker Proving Broker started
16:47:24 [16:47:24.809]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed5bae7b802a6ee9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed5bae7b802a6ee9"}
16:47:24 [16:47:24.810]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyvzQyy/1 with map size 1073741824
16:47:24 [16:47:24.812]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:24 [16:47:24.935]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ed5bae7b802a6ee9 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:ed5bae7b802a6ee9"}
16:47:24 [16:47:24.941]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11d34ccdd07683ad epochNumber=10
{"provingJobId":"1:BASE_PARITY:11d34ccdd07683ad"}
16:47:24 [16:47:24.943]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testyvzQyy/10 with map size 1073741824
16:47:24 [16:47:24.946]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:24 [16:47:24.986]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:24 [16:47:24.987]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:47:25 [16:47:25.092]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.093]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f09c27f0bec43a88 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f09c27f0bec43a88"}
16:47:25 [16:47:25.094]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZZ1G18/1 with map size 1073741824
16:47:25 [16:47:25.095]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.270]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f09c27f0bec43a88 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f09c27f0bec43a88"}
16:47:25 [16:47:25.321]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.322]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a4792eac9aefe5b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a4792eac9aefe5b6"}
16:47:25 [16:47:25.323]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testczyJiP/1 with map size 1073741824
16:47:25 [16:47:25.324]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.340]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a4792eac9aefe5b6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a4792eac9aefe5b6"}
16:47:25 [16:47:25.344]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.345]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1632a7aea7096d5a epochNumber=1
{"provingJobId":"1:BASE_PARITY:1632a7aea7096d5a"}
16:47:25 [16:47:25.345]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMEP7DB/1 with map size 1073741824
16:47:25 [16:47:25.347]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.364]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1632a7aea7096d5a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1632a7aea7096d5a"}
16:47:25 [16:47:25.365]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1632a7aea7096d5a type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:1632a7aea7096d5a"}
16:47:25 [16:47:25.374]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.375]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:07513018d8a7ad84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:07513018d8a7ad84"}
16:47:25 [16:47:25.376]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPqFiwC/1 with map size 1073741824
16:47:25 [16:47:25.385]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.405]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:07513018d8a7ad84 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:07513018d8a7ad84"}
16:47:25 [16:47:25.416]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3474f71f627e8c76 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3474f71f627e8c76"}
16:47:25 [16:47:25.417]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWNfoeK/1 with map size 1073741824
16:47:25 [16:47:25.420]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.438]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3474f71f627e8c76 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3474f71f627e8c76"}
16:47:25 [16:47:25.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:297a46682e22e263 epochNumber=3
{"provingJobId":"1:BASE_PARITY:297a46682e22e263"}
16:47:25 [16:47:25.440]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testWNfoeK/3 with map size 1073741824
16:47:25 [16:47:25.441]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.467]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:25 [16:47:25.468]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:47:25 [16:47:25.486]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3474f71f627e8c76 err=test error
{"provingJoId":"1:BASE_PARITY:3474f71f627e8c76"}
16:47:25 [16:47:25.496]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testq7R1kD/1 with map size 1073741824
16:47:25 [16:47:25.500]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.522]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testq7R1kD/2 with map size 1073741824
16:47:25 [16:47:25.524]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.543]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.545]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8c155e3fcf054c23 settled=false
{"provingJobId":"1:BASE_PARITY:8c155e3fcf054c23","status":"pending"}
16:47:25 [16:47:25.547]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e30009da23c56baa settled=false
{"provingJobId":"1:BASE_PARITY:e30009da23c56baa","status":"pending"}
16:47:25 [16:47:25.553]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjbylfy/1 with map size 1073741824
16:47:25 [16:47:25.556]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.575]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjbylfy/2 with map size 1073741824
16:47:25 [16:47:25.591]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.624]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.627]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:198a0de5aa767ca6 settled=true
{"provingJobId":"1:BASE_PARITY:198a0de5aa767ca6","status":"fulfilled"}
16:47:25 [16:47:25.628]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:3eedfb4e0be24239 settled=true
{"provingJobId":"2:BASE_PARITY:3eedfb4e0be24239","status":"fulfilled"}
16:47:25 [16:47:25.636]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlrG1Zn/1 with map size 1073741824
16:47:25 [16:47:25.640]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.674]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlrG1Zn/2 with map size 1073741824
16:47:25 [16:47:25.691]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.716]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.718]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:08b8e4640898ffaa settled=true
{"provingJobId":"1:BASE_PARITY:08b8e4640898ffaa","status":"fulfilled"}
16:47:25 [16:47:25.721]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2fa0164fee1fdc2c settled=false
{"provingJobId":"1:BASE_PARITY:2fa0164fee1fdc2c","status":"pending"}
16:47:25 [16:47:25.726]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ff6afef5b79189f epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ff6afef5b79189f"}
16:47:25 [16:47:25.728]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLYasTM/1 with map size 1073741824
16:47:25 [16:47:25.731]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.755]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.756]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4dcdb42457fc7fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4dcdb42457fc7fa"}
16:47:25 [16:47:25.757]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:d4dcdb42457fc7fa: Error: db error: Error: db error
16:47:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:d4dcdb42457fc7fa"}
16:47:25 [16:47:25.762]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da8127d20fff83ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:da8127d20fff83ac"}
16:47:25 [16:47:25.763]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqkeo1j/1 with map size 1073741824
16:47:25 [16:47:25.768]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.797]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:da8127d20fff83ac type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:da8127d20fff83ac"}
16:47:25 [16:47:25.798]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:da8127d20fff83ac type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:da8127d20fff83ac"}
16:47:25 [16:47:25.812]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc7148eb1a3e1490 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bc7148eb1a3e1490"}
16:47:25 [16:47:25.814]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testC0kZvV/1 with map size 1073741824
16:47:25 [16:47:25.816]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.838]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bc7148eb1a3e1490 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bc7148eb1a3e1490"}
16:47:25 [16:47:25.838]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bc7148eb1a3e1490 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bc7148eb1a3e1490"}
16:47:25 [16:47:25.839]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:bc7148eb1a3e1490: Error: db error
16:47:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:bc7148eb1a3e1490"}
16:47:25 [16:47:25.847]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.848]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b40531361c89296 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b40531361c89296"}
16:47:25 [16:47:25.849]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test31InlO/1 with map size 1073741824
16:47:25 [16:47:25.851]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.872]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4b40531361c89296 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4b40531361c89296"}
16:47:25 [16:47:25.873]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4b40531361c89296 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4b40531361c89296"}
16:47:25 [16:47:25.890]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.891]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f24ff17ea954fa9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f24ff17ea954fa9"}
16:47:25 [16:47:25.893]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLQKZx7/1 with map size 1073741824
16:47:25 [16:47:25.897]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.915]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1f24ff17ea954fa9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1f24ff17ea954fa9"}
16:47:25 [16:47:25.915]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1f24ff17ea954fa9 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1f24ff17ea954fa9"}
16:47:25 [16:47:25.916]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:1f24ff17ea954fa9 jobErr=test error: Error: db error
16:47:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:1f24ff17ea954fa9"}
16:47:25 [16:47:25.919]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.920]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:55026183ca8e5b41 not found
{"provingJobId":"1:BASE_PARITY:55026183ca8e5b41"}
16:47:25 [16:47:25.926]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.927]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d1b193d4ff3ca067 err=test error
{"provingJoId":"1:BASE_PARITY:d1b193d4ff3ca067"}
16:47:25 [16:47:25.932]
INFO:
prover-client:proving-broker Proving Broker started
16:47:25 [16:47:25.943]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c5c4f368ab9f79e epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c5c4f368ab9f79e"}
16:47:25 [16:47:25.945]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testu9SmMD/1 with map size 1073741824
16:47:25 [16:47:25.949]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:25 [16:47:25.970]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0c5c4f368ab9f79e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0c5c4f368ab9f79e"}
16:47:25 [16:47:25.971]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0c5c4f368ab9f79e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0c5c4f368ab9f79e"}
16:47:25 [16:47:25.987]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:ce3c92b67460f30e epochNumber=2
{"provingJobId":"2:BASE_PARITY:ce3c92b67460f30e"}
16:47:25 [16:47:25.989]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testu9SmMD/2 with map size 1073741824
16:47:25 [16:47:25.993]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:26 [16:47:26.015]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:ce3c92b67460f30e type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:ce3c92b67460f30e"}
16:47:26 [16:47:26.015]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:ce3c92b67460f30e type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:ce3c92b67460f30e"}
16:47:26 [16:47:26.084]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:bbd2ba2d5dc9f6a8 epochNumber=3
{"provingJobId":"3:BASE_PARITY:bbd2ba2d5dc9f6a8"}
16:47:26 [16:47:26.092]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testu9SmMD/3 with map size 1073741824
16:47:26 [16:47:26.103]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:26 [16:47:26.136]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:26 [16:47:26.137]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:47:26 [16:47:26.184]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:6b54ff577bbbadf6 epochNumber=4
{"provingJobId":"4:BASE_PARITY:6b54ff577bbbadf6"}
16:47:26 [16:47:26.189]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testu9SmMD/4 with map size 1073741824
16:47:26 [16:47:26.198]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:26 [16:47:26.243]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:26 [16:47:26.243]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
16:47:26 [16:47:26.285]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:d65cfbef7a2e387d epochNumber=5
{"provingJobId":"5:BASE_PARITY:d65cfbef7a2e387d"}
16:47:26 [16:47:26.287]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testu9SmMD/5 with map size 1073741824
16:47:26 [16:47:26.289]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:47:26 [16:47:26.323]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:47:26 [16:47:26.324]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
16:47:26
PASS src/proving_broker/proving_broker.test.ts (
13.233 s)
16:47:26 ProvingBroker
16:47:26 Producer API
16:47:26
✓ refuses stale jobs (43 ms)
16:47:26
✓ enqueues jobs (6 ms)
16:47:26
✓ ignores duplicate jobs (5 ms)
16:47:26
✓ reports correct status when enqueuing repeat jobs (5 ms)
16:47:26
✓ throws an error in case of duplicate job IDs (23 ms)
16:47:26
✓ returns not-found status for non-existing jobs (2 ms)
16:47:26
✓ cancels jobs in queue (3 ms)
16:47:26
✓ cancels jobs in-progress (2 ms)
16:47:26
✓ returns job result if successful (3 ms)
16:47:26
✓ returns job error if failed (2 ms)
16:47:26
✓ correctly returns job status for concurrent writes (3 ms)
16:47:26
✓ refuses stale jobs (34 ms)
16:47:26
✓ enqueues jobs (31 ms)
16:47:26
✓ ignores duplicate jobs (44 ms)
16:47:26
✓ reports correct status when enqueuing repeat jobs (30 ms)
16:47:26
✓ throws an error in case of duplicate job IDs (25 ms)
16:47:26
✓ returns not-found status for non-existing jobs (3 ms)
16:47:26
✓ cancels jobs in queue (32 ms)
16:47:26
✓ cancels jobs in-progress (32 ms)
16:47:26
✓ returns job result if successful (34 ms)
16:47:26
✓ returns job error if failed (46 ms)
16:47:26
✓ correctly returns job status for concurrent writes (42 ms)
16:47:26 Consumer API
16:47:26
✓ returns undefined if no jobs are available (1 ms)
16:47:26
✓ returns jobs in priority order (1 ms)
16:47:26
✓ returns undefined if no jobs are available for the given allowList (1 ms)
16:47:26
✓ returns a job if it is in the allowList (4 ms)
16:47:26
✓ returns the most important job if it is in the allowList (3 ms)
16:47:26
✓ returns any job if filter is empty (2 ms)
16:47:26
✓ returns a new job when reporting job success (3 ms)
16:47:26
✓ returns a new job when reporting permanent error (2 ms)
16:47:26
✓ returns a new job when reporting retry-able error (2 ms)
16:47:26
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
16:47:26
✓ returns a new job if job is already in progress elsewhere (1062 ms)
16:47:26
✓ avoids sending the same job to a new agent after a restart (1007 ms)
16:47:26
✓ avoids sending a completed job to a new agent after a restart (1009 ms)
16:47:26
✓ tracks job result if in progress (5 ms)
16:47:26
✓ tracks job result even if job is in queue (3 ms)
16:47:26
✓ ignores reported job error if unknown job (2 ms)
16:47:26
✓ ignores job result if unknown job (2 ms)
16:47:26
✓ returns undefined if no jobs are available (2 ms)
16:47:26
✓ returns jobs in priority order (64 ms)
16:47:26
✓ returns undefined if no jobs are available for the given allowList (27 ms)
16:47:26
✓ returns a job if it is in the allowList (63 ms)
16:47:26
✓ returns the most important job if it is in the allowList (54 ms)
16:47:26
✓ returns any job if filter is empty (60 ms)
16:47:26
✓ returns a new job when reporting job success (41 ms)
16:47:26
✓ returns a new job when reporting permanent error (35 ms)
16:47:26
✓ returns a new job when reporting retry-able error (37 ms)
16:47:26
✓ returns a new job when reporting progress if current one is cancelled (35 ms)
16:47:26
✓ returns a new job if job is already in progress elsewhere (1112 ms)
16:47:26
✓ avoids sending the same job to a new agent after a restart (1065 ms)
16:47:26
✓ avoids sending a completed job to a new agent after a restart (1061 ms)
16:47:26
✓ tracks job result if in progress (59 ms)
16:47:26
✓ tracks job result even if job is in queue (67 ms)
16:47:26
✓ ignores reported job error if unknown job (3 ms)
16:47:26
✓ ignores job result if unknown job (5 ms)
16:47:26 Timeouts
16:47:26
✓ tracks in progress jobs (1 ms)
16:47:26
✓ re-enqueues jobs that time out (107 ms)
16:47:26
✓ cancel stale jobs that time out (235 ms)
16:47:26
✓ keeps the jobs in progress while it is alive (202 ms)
16:47:26
✓ tracks in progress jobs (23 ms)
16:47:26
✓ re-enqueues jobs that time out (131 ms)
16:47:26
✓ cancel stale jobs that time out (284 ms)
16:47:26
✓ keeps the jobs in progress while it is alive (229 ms)
16:47:26 Retries
16:47:26
✓ retries jobs (2 ms)
16:47:26
✓ retries up to a maximum number of times (3 ms)
16:47:26
✓ passing retry=false does not retry (2 ms)
16:47:26
✓ does not retry if job is stale (30 ms)
16:47:26
✓ retries jobs (22 ms)
16:47:26
✓ retries up to a maximum number of times (29 ms)
16:47:26
✓ passing retry=false does not retry (40 ms)
16:47:26
✓ does not retry if job is stale (75 ms)
16:47:26 Database management
16:47:26
✓ re-enqueues proof requests on start (3 ms)
16:47:26
✓ restores proof results on start (1 ms)
16:47:26
✓ only re-enqueues unfinished jobs (1 ms)
16:47:26
✓ saves job when enqueued (3 ms)
16:47:26
✓ does not retain job if database fails to save (2 ms)
16:47:26
✓ saves job result (1 ms)
16:47:26
✓ saves result even if database fails to save (2 ms)
16:47:26
✓ saves job error (2 ms)
16:47:26
✓ saves job error even if database fails to save (2 ms)
16:47:26
✓ does not save job result if job is unknown (4 ms)
16:47:26
✓ does not save job error if job is unknown (1 ms)
16:47:26
✓ cleans up old jobs periodically (265 ms)
16:47:26
✓ re-enqueues proof requests on start (61 ms)
16:47:26
✓ restores proof results on start (81 ms)
16:47:26
✓ only re-enqueues unfinished jobs (93 ms)
16:47:26
✓ saves job when enqueued (27 ms)
16:47:26
✓ does not retain job if database fails to save (8 ms)
16:47:26
✓ saves job result (47 ms)
16:47:26
✓ saves result even if database fails to save (33 ms)
16:47:26
✓ saves job error (45 ms)
16:47:26
✓ saves job error even if database fails to save (29 ms)
16:47:26
✓ does not save job result if job is unknown (3 ms)
16:47:26
✓ does not save job error if job is unknown (8 ms)
16:47:26
✓ cleans up old jobs periodically (439 ms)
16:47:26
16:47:26
Test Suites: 1 passed, 1 total
16:47:26
Tests: 96 passed, 96 total
16:47:26
Snapshots: 0 total
16:47:26
Time: 13.364 s
16:47:26
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
16:47:26
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?