Parent Log:
http://ci.aztec-labs.com/45dd6f579f0176e2
Command: 4174e5df391694d3 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/8d254a16364bbc8136cad6732cc978ea1a895cae
Env: REF_NAME=gh-readonly-queue/next/pr-15073-7763eef5eb7f045b9184a0a4591b394296eb3aa8 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 16 15:14:58 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15073_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
15:15:03 [15:15:03.233]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.241]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06939b93887d7d60 epochNumber=42
{"provingJobId":"1:BASE_PARITY:06939b93887d7d60"}
15:15:03 [15:15:03.245]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:f86a6eaad61392e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f86a6eaad61392e2"}
15:15:03 [15:15:03.286]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.287]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b0977c033d667cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b0977c033d667cb"}
15:15:03 [15:15:03.288]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:591e821a09b5900c epochNumber=1
{"provingJobId":"1:BASE_PARITY:591e821a09b5900c"}
15:15:03 [15:15:03.290]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.291]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7cda276ac0838629 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7cda276ac0838629"}
15:15:03 [15:15:03.292]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7cda276ac0838629 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7cda276ac0838629"}
15:15:03 [15:15:03.295]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.296]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e13b21957ea7f840 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e13b21957ea7f840"}
15:15:03 [15:15:03.297]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e13b21957ea7f840 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e13b21957ea7f840"}
15:15:03 [15:15:03.297]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e13b21957ea7f840 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e13b21957ea7f840"}
15:15:03 [15:15:03.298]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e13b21957ea7f840 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e13b21957ea7f840"}
15:15:03 [15:15:03.299]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.299]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:761fb14c9753dd75 epochNumber=1
{"provingJobId":"1:BASE_PARITY:761fb14c9753dd75"}
15:15:03 [15:15:03.322]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.324]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.325]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c8a0bab67b4b581c epochNumber=1
{"provingJobId":"1:BASE_PARITY:c8a0bab67b4b581c"}
15:15:03 [15:15:03.325]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c8a0bab67b4b581c
{"provingJobId":"1:BASE_PARITY:c8a0bab67b4b581c"}
15:15:03 [15:15:03.326]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c8a0bab67b4b581c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c8a0bab67b4b581c"}
15:15:03 [15:15:03.326]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c8a0bab67b4b581c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c8a0bab67b4b581c"}
15:15:03 [15:15:03.327]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.328]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66c6cdec060f553e epochNumber=1
{"provingJobId":"1:BASE_PARITY:66c6cdec060f553e"}
15:15:03 [15:15:03.328]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:66c6cdec060f553e
{"provingJobId":"1:BASE_PARITY:66c6cdec060f553e"}
15:15:03 [15:15:03.328]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:66c6cdec060f553e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:66c6cdec060f553e"}
15:15:03 [15:15:03.330]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87c72337042b3020 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87c72337042b3020"}
15:15:03 [15:15:03.331]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:87c72337042b3020 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:87c72337042b3020"}
15:15:03 [15:15:03.331]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:87c72337042b3020 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:87c72337042b3020"}
15:15:03 [15:15:03.342]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.343]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e6172865b7e950d8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e6172865b7e950d8"}
15:15:03 [15:15:03.343]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e6172865b7e950d8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e6172865b7e950d8"}
15:15:03 [15:15:03.344]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e6172865b7e950d8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e6172865b7e950d8"}
15:15:03 [15:15:03.345]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.345]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df9755dd1603f1a5 epochNumber=0
{"provingJobId":"1:BASE_PARITY:df9755dd1603f1a5"}
15:15:03 [15:15:03.346]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:df9755dd1603f1a5 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:df9755dd1603f1a5"}
15:15:03 [15:15:03.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53472798da918f61 epochNumber=0
{"provingJobId":"1:BASE_PARITY:53472798da918f61"}
15:15:03 [15:15:03.346]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:df9755dd1603f1a5 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:df9755dd1603f1a5"}
15:15:03 [15:15:03.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1436c41cbd383d27 epochNumber=0
{"provingJobId":"1:BASE_PARITY:1436c41cbd383d27"}
15:15:03 [15:15:03.348]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.349]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.350]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9384cbe584a5bb6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e9384cbe584a5bb6"}
15:15:03 [15:15:03.350]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26f8b9c7aec04b5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:26f8b9c7aec04b5d"}
15:15:03 [15:15:03.351]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.352]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d4541bda2e81575 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d4541bda2e81575"}
15:15:03 [15:15:03.353]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.353]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d16da0b0825f5412 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d16da0b0825f5412"}
15:15:03 [15:15:03.353]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0268160b736c77dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:0268160b736c77dc"}
15:15:03 [15:15:03.354]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:864bf6156817f464 epochNumber=2
{"provingJobId":"1:BASE_PARITY:864bf6156817f464"}
15:15:03 [15:15:03.354]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e022f17c67338bfa epochNumber=1
{"provingJobId":"1:BASE_PARITY:e022f17c67338bfa"}
15:15:03 [15:15:03.355]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:446c981353eb4986 epochNumber=1
{"provingJobId":"1:BASE_PARITY:446c981353eb4986"}
15:15:03 [15:15:03.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26c8653c2f2a8098 epochNumber=1
{"provingJobId":"1:BASE_PARITY:26c8653c2f2a8098"}
15:15:03 [15:15:03.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89734caab2fd32b9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:89734caab2fd32b9"}
15:15:03 [15:15:03.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:861eabbc7958ba0d epochNumber=1
{"provingJobId":"1:BASE_PARITY:861eabbc7958ba0d"}
15:15:03 [15:15:03.358]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.359]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93cef52d176b6d79 epochNumber=1
{"provingJobId":"1:BASE_PARITY:93cef52d176b6d79"}
15:15:03 [15:15:03.359]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ed19a98019ec13e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ed19a98019ec13e"}
15:15:03 [15:15:03.359]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55e045f47a3e9820 epochNumber=2
{"provingJobId":"1:BASE_PARITY:55e045f47a3e9820"}
15:15:03 [15:15:03.360]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:700aada63dfc9bd7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:700aada63dfc9bd7"}
15:15:03 [15:15:03.361]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.361]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76df6a820ef5e229 epochNumber=1
{"provingJobId":"1:BASE_PARITY:76df6a820ef5e229"}
15:15:03 [15:15:03.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c46d1c60f284e0a epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c46d1c60f284e0a"}
15:15:03 [15:15:03.362]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:76df6a820ef5e229 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:76df6a820ef5e229"}
15:15:03 [15:15:03.363]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.364]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81b2ea52f67d09fc epochNumber=1
{"provingJobId":"1:BASE_PARITY:81b2ea52f67d09fc"}
15:15:03 [15:15:03.364]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c429a3aaa115dca epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c429a3aaa115dca"}
15:15:03 [15:15:03.364]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:81b2ea52f67d09fc type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:81b2ea52f67d09fc"}
15:15:03 [15:15:03.365]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9434778df3a99f6b epochNumber=1
{"provingJobId":"1:BASE_PARITY:9434778df3a99f6b"}
15:15:03 [15:15:03.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d521cd4220c41d5c epochNumber=1
{"provingJobId":"1:BASE_PARITY:d521cd4220c41d5c"}
15:15:03 [15:15:03.367]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9434778df3a99f6b type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:9434778df3a99f6b"}
15:15:03 [15:15:03.368]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.368]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ecae0ee60e9f61a epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ecae0ee60e9f61a"}
15:15:03 [15:15:03.368]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:8ecae0ee60e9f61a
{"provingJobId":"1:BASE_PARITY:8ecae0ee60e9f61a"}
15:15:03 [15:15:03.369]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8ecae0ee60e9f61a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:8ecae0ee60e9f61a"}
15:15:03 [15:15:03.369]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5f2d834f86289be epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5f2d834f86289be"}
15:15:03 [15:15:03.370]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8ecae0ee60e9f61a has already been completed
{"provingJobId":"1:BASE_PARITY:8ecae0ee60e9f61a"}
15:15:03 [15:15:03.371]
INFO:
prover-client:proving-broker Proving Broker started
15:15:03 [15:15:03.371]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:064e1f5f9e8c5c0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:064e1f5f9e8c5c0b"}
15:15:03 [15:15:03.372]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ee621d68484f814 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0ee621d68484f814"}
15:15:04 [15:15:04.423]
INFO:
prover-client:proving-broker Proving Broker started
15:15:04 [15:15:04.423]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:064e1f5f9e8c5c0b settled=false
{"provingJobId":"1:BASE_PARITY:064e1f5f9e8c5c0b","status":"pending"}
15:15:04 [15:15:04.424]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0ee621d68484f814 settled=false
{"provingJobId":"1:BASE_PARITY:0ee621d68484f814","status":"pending"}
15:15:04 [15:15:04.424]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:064e1f5f9e8c5c0b type=BASE_PARITY startedAt=1750086903372 older agent has taken job
{"provingJobId":"1:BASE_PARITY:064e1f5f9e8c5c0b"}
15:15:04 [15:15:04.425]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:064e1f5f9e8c5c0b type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:064e1f5f9e8c5c0b"}
15:15:04 [15:15:04.426]
INFO:
prover-client:proving-broker Proving Broker started
15:15:04 [15:15:04.426]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44a720e8c73e32df epochNumber=1
{"provingJobId":"1:BASE_PARITY:44a720e8c73e32df"}
15:15:04 [15:15:04.426]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:040da29d6cc69b1b epochNumber=2
{"provingJobId":"1:BASE_PARITY:040da29d6cc69b1b"}
15:15:05 [15:15:05.427]
INFO:
prover-client:proving-broker Proving Broker started
15:15:05 [15:15:05.428]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:44a720e8c73e32df settled=false
{"provingJobId":"1:BASE_PARITY:44a720e8c73e32df","status":"pending"}
15:15:05 [15:15:05.428]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:040da29d6cc69b1b settled=false
{"provingJobId":"1:BASE_PARITY:040da29d6cc69b1b","status":"pending"}
15:15:05 [15:15:05.429]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:44a720e8c73e32df type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:44a720e8c73e32df"}
15:15:05 [15:15:05.430]
INFO:
prover-client:proving-broker Proving Broker started
15:15:05 [15:15:05.431]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1922f4204b55a40 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1922f4204b55a40"}
15:15:05 [15:15:05.431]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ad762fdb144aefd epochNumber=2
{"provingJobId":"1:BASE_PARITY:2ad762fdb144aefd"}
15:15:06 [15:15:06.432]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.433]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a1922f4204b55a40 settled=false
{"provingJobId":"1:BASE_PARITY:a1922f4204b55a40","status":"pending"}
15:15:06 [15:15:06.433]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2ad762fdb144aefd settled=false
{"provingJobId":"1:BASE_PARITY:2ad762fdb144aefd","status":"pending"}
15:15:06 [15:15:06.434]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a1922f4204b55a40 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a1922f4204b55a40"}
15:15:06 [15:15:06.434]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a1922f4204b55a40 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a1922f4204b55a40"}
15:15:06 [15:15:06.436]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.436]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c76a0c9e7ba3f0c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c76a0c9e7ba3f0c7"}
15:15:06 [15:15:06.436]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3127e5e7fd5f0659 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3127e5e7fd5f0659"}
15:15:06 [15:15:06.437]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c76a0c9e7ba3f0c7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c76a0c9e7ba3f0c7"}
15:15:06 [15:15:06.438]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3127e5e7fd5f0659 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3127e5e7fd5f0659"}
15:15:06 [15:15:06.439]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c9abe1af341e527 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c9abe1af341e527"}
15:15:06 [15:15:06.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1e25d7cb2955bd4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c1e25d7cb2955bd4"}
15:15:06 [15:15:06.440]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6c9abe1af341e527 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6c9abe1af341e527"}
15:15:06 [15:15:06.440]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6c9abe1af341e527 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6c9abe1af341e527"}
15:15:06 [15:15:06.440]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c1e25d7cb2955bd4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c1e25d7cb2955bd4"}
15:15:06 [15:15:06.441]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.442]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:22fe3c4b10be58e8 err=test error
{"provingJoId":"1:BASE_PARITY:22fe3c4b10be58e8"}
15:15:06 [15:15:06.443]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.444]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d7f7c3c0765a334e not found
{"provingJobId":"1:BASE_PARITY:d7f7c3c0765a334e"}
15:15:06 [15:15:06.445]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.445]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ecce7905d2b6ebdc epochNumber=1
{"provingJobId":"1:BASE_PARITY:ecce7905d2b6ebdc"}
15:15:06 [15:15:06.447]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8cffdf8fe4601ddf epochNumber=1
{"provingJobId":"1:BASE_PARITY:8cffdf8fe4601ddf"}
15:15:06 [15:15:06.550]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8cffdf8fe4601ddf timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8cffdf8fe4601ddf"}
15:15:06 [15:15:06.556]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.556]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:38aa399935490a7e epochNumber=1
{"provingJobId":"1:BASE_PARITY:38aa399935490a7e"}
15:15:06 [15:15:06.658]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:38aa399935490a7e timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:38aa399935490a7e"}
15:15:06 [15:15:06.665]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:706c6fb4d450dc6d epochNumber=10
{"provingJobId":"1:BASE_PARITY:706c6fb4d450dc6d"}
15:15:06 [15:15:06.684]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:06 [15:15:06.792]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.793]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b3223a09e147da2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b3223a09e147da2"}
15:15:06 [15:15:06.944]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1b3223a09e147da2 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1b3223a09e147da2"}
15:15:06 [15:15:06.995]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.996]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1fd6e5a0a626e346 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1fd6e5a0a626e346"}
15:15:06 [15:15:06.997]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1fd6e5a0a626e346 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1fd6e5a0a626e346"}
15:15:06 [15:15:06.998]
INFO:
prover-client:proving-broker Proving Broker started
15:15:06 [15:15:06.998]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e53cf2454401e11 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e53cf2454401e11"}
15:15:06 [15:15:06.999]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5e53cf2454401e11 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:5e53cf2454401e11"}
15:15:06 [15:15:06.999]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5e53cf2454401e11 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:5e53cf2454401e11"}
15:15:07 [15:15:07.000]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.001]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a97f2f0320c8e140 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a97f2f0320c8e140"}
15:15:07 [15:15:07.001]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a97f2f0320c8e140 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a97f2f0320c8e140"}
15:15:07 [15:15:07.002]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.003]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3aa7a3776d095d31 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3aa7a3776d095d31"}
15:15:07 [15:15:07.003]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3aa7a3776d095d31 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3aa7a3776d095d31"}
15:15:07 [15:15:07.004]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f62ce72b903f79b epochNumber=3
{"provingJobId":"1:BASE_PARITY:3f62ce72b903f79b"}
15:15:07 [15:15:07.028]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:07 [15:15:07.030]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3aa7a3776d095d31 err=test error
{"provingJoId":"1:BASE_PARITY:3aa7a3776d095d31"}
15:15:07 [15:15:07.031]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.031]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5f93e23277769ce9 settled=false
{"provingJobId":"1:BASE_PARITY:5f93e23277769ce9","status":"pending"}
15:15:07 [15:15:07.032]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6793a29602797278 settled=false
{"provingJobId":"1:BASE_PARITY:6793a29602797278","status":"pending"}
15:15:07 [15:15:07.033]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.034]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:12bd338c789a7f69 settled=true
{"provingJobId":"1:BASE_PARITY:12bd338c789a7f69","status":"fulfilled"}
15:15:07 [15:15:07.034]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:00eca15cd9a85019 settled=true
{"provingJobId":"2:BASE_PARITY:00eca15cd9a85019","status":"fulfilled"}
15:15:07 [15:15:07.035]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.035]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6fa891b0ab4789fa settled=true
{"provingJobId":"1:BASE_PARITY:6fa891b0ab4789fa","status":"fulfilled"}
15:15:07 [15:15:07.036]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a80ddd64a369ed05 settled=false
{"provingJobId":"1:BASE_PARITY:a80ddd64a369ed05","status":"pending"}
15:15:07 [15:15:07.037]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.038]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:07f8ef51d399ace5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:07f8ef51d399ace5"}
15:15:07 [15:15:07.039]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.040]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f34dccbd4f8c238 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f34dccbd4f8c238"}
15:15:07 [15:15:07.040]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:9f34dccbd4f8c238: Error: db error: Error: db error
15:15:07 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:9f34dccbd4f8c238"}
15:15:07 [15:15:07.041]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.041]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b4fd2683cb7fb28b epochNumber=1
{"provingJobId":"1:BASE_PARITY:b4fd2683cb7fb28b"}
15:15:07 [15:15:07.042]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b4fd2683cb7fb28b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b4fd2683cb7fb28b"}
15:15:07 [15:15:07.042]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b4fd2683cb7fb28b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b4fd2683cb7fb28b"}
15:15:07 [15:15:07.043]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.043]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cceebb72eff058ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:cceebb72eff058ad"}
15:15:07 [15:15:07.044]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cceebb72eff058ad type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:cceebb72eff058ad"}
15:15:07 [15:15:07.044]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cceebb72eff058ad type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cceebb72eff058ad"}
15:15:07 [15:15:07.044]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:cceebb72eff058ad: Error: db error
15:15:07 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:cceebb72eff058ad"}
15:15:07 [15:15:07.045]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.046]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7bdbd2944d43e060 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7bdbd2944d43e060"}
15:15:07 [15:15:07.046]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7bdbd2944d43e060 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7bdbd2944d43e060"}
15:15:07 [15:15:07.046]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7bdbd2944d43e060 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7bdbd2944d43e060"}
15:15:07 [15:15:07.047]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.047]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ad3358d56a0027c epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ad3358d56a0027c"}
15:15:07 [15:15:07.047]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5ad3358d56a0027c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5ad3358d56a0027c"}
15:15:07 [15:15:07.048]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5ad3358d56a0027c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5ad3358d56a0027c"}
15:15:07 [15:15:07.048]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:5ad3358d56a0027c jobErr=test error: Error: db error
15:15:07 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:5ad3358d56a0027c"}
15:15:07 [15:15:07.049]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.049]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:58e33074e5af6073 not found
{"provingJobId":"1:BASE_PARITY:58e33074e5af6073"}
15:15:07 [15:15:07.050]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.051]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8b32e56585821249 err=test error
{"provingJoId":"1:BASE_PARITY:8b32e56585821249"}
15:15:07 [15:15:07.051]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.062]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4cccc6d2c5e14de5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4cccc6d2c5e14de5"}
15:15:07 [15:15:07.063]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4cccc6d2c5e14de5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4cccc6d2c5e14de5"}
15:15:07 [15:15:07.063]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4cccc6d2c5e14de5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4cccc6d2c5e14de5"}
15:15:07 [15:15:07.073]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:1f4e0cdcae6724bf epochNumber=2
{"provingJobId":"2:BASE_PARITY:1f4e0cdcae6724bf"}
15:15:07 [15:15:07.074]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:1f4e0cdcae6724bf type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:1f4e0cdcae6724bf"}
15:15:07 [15:15:07.074]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:1f4e0cdcae6724bf type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:1f4e0cdcae6724bf"}
15:15:07 [15:15:07.135]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:3fb883ab486cc6c5 epochNumber=3
{"provingJobId":"3:BASE_PARITY:3fb883ab486cc6c5"}
15:15:07 [15:15:07.154]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:07 [15:15:07.196]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:45f025bf6af0d038 epochNumber=4
{"provingJobId":"4:BASE_PARITY:45f025bf6af0d038"}
15:15:07 [15:15:07.204]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:07 [15:15:07.258]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:6792f33ded4d77f4 epochNumber=5
{"provingJobId":"5:BASE_PARITY:6792f33ded4d77f4"}
15:15:07 [15:15:07.280]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:07 [15:15:07.312]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.313]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6fd0fdd129119a5 epochNumber=42
{"provingJobId":"1:BASE_PARITY:a6fd0fdd129119a5"}
15:15:07 [15:15:07.314]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testqdye6H/42 with map size 1073741824
15:15:07 [15:15:07.317]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.339]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:218e7f1a565cad4c epochNumber=1
{"provingJobId":"1:BASE_PARITY:218e7f1a565cad4c"}
15:15:07 [15:15:07.344]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.345]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e89a11f78611ef3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e89a11f78611ef3"}
15:15:07 [15:15:07.345]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRoJ5YY/1 with map size 1073741824
15:15:07 [15:15:07.348]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.376]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:661b200e7bbdb99d epochNumber=1
{"provingJobId":"1:BASE_PARITY:661b200e7bbdb99d"}
15:15:07 [15:15:07.389]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.389]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d4f2818cd5c68ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d4f2818cd5c68ba"}
15:15:07 [15:15:07.390]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVTM0qE/1 with map size 1073741824
15:15:07 [15:15:07.394]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.416]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6d4f2818cd5c68ba epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6d4f2818cd5c68ba"}
15:15:07 [15:15:07.421]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.422]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88cb872e5eb3369b epochNumber=1
{"provingJobId":"1:BASE_PARITY:88cb872e5eb3369b"}
15:15:07 [15:15:07.422]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3xHw2E/1 with map size 1073741824
15:15:07 [15:15:07.424]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.441]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:88cb872e5eb3369b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:88cb872e5eb3369b"}
15:15:07 [15:15:07.441]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:88cb872e5eb3369b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:88cb872e5eb3369b"}
15:15:07 [15:15:07.447]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:88cb872e5eb3369b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:88cb872e5eb3369b"}
15:15:07 [15:15:07.450]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.450]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af8f19ee986c2508 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af8f19ee986c2508"}
15:15:07 [15:15:07.450]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMCCjy7/1 with map size 1073741824
15:15:07 [15:15:07.452]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.472]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.479]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.480]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77daa6e84097476e epochNumber=1
{"provingJobId":"1:BASE_PARITY:77daa6e84097476e"}
15:15:07 [15:15:07.480]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJXq6G4/1 with map size 1073741824
15:15:07 [15:15:07.490]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.507]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:77daa6e84097476e
{"provingJobId":"1:BASE_PARITY:77daa6e84097476e"}
15:15:07 [15:15:07.507]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:77daa6e84097476e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:77daa6e84097476e"}
15:15:07 [15:15:07.508]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:77daa6e84097476e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:77daa6e84097476e"}
15:15:07 [15:15:07.516]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.517]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b72d1f0786b1bd6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b72d1f0786b1bd6"}
15:15:07 [15:15:07.518]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwMZWQC/1 with map size 1073741824
15:15:07 [15:15:07.519]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.536]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9b72d1f0786b1bd6
{"provingJobId":"1:BASE_PARITY:9b72d1f0786b1bd6"}
15:15:07 [15:15:07.536]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9b72d1f0786b1bd6 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:9b72d1f0786b1bd6"}
15:15:07 [15:15:07.544]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.545]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:acdf3eb38bfbe36d epochNumber=1
{"provingJobId":"1:BASE_PARITY:acdf3eb38bfbe36d"}
15:15:07 [15:15:07.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testExOOs4/1 with map size 1073741824
15:15:07 [15:15:07.547]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.564]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:acdf3eb38bfbe36d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:acdf3eb38bfbe36d"}
15:15:07 [15:15:07.564]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:acdf3eb38bfbe36d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:acdf3eb38bfbe36d"}
15:15:07 [15:15:07.572]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.573]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ff06254e4cc4bb7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0ff06254e4cc4bb7"}
15:15:07 [15:15:07.573]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDHZS6m/1 with map size 1073741824
15:15:07 [15:15:07.575]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.596]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0ff06254e4cc4bb7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0ff06254e4cc4bb7"}
15:15:07 [15:15:07.596]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0ff06254e4cc4bb7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0ff06254e4cc4bb7"}
15:15:07 [15:15:07.605]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:966895ffda1121c8 epochNumber=0
{"provingJobId":"1:BASE_PARITY:966895ffda1121c8"}
15:15:07 [15:15:07.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testxO2qwV/0 with map size 1073741824
15:15:07 [15:15:07.609]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.625]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:966895ffda1121c8 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:966895ffda1121c8"}
15:15:07 [15:15:07.626]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7cfd1bbc56be9eb epochNumber=0
{"provingJobId":"1:BASE_PARITY:b7cfd1bbc56be9eb"}
15:15:07 [15:15:07.626]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:966895ffda1121c8 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:966895ffda1121c8"}
15:15:07 [15:15:07.627]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c05b785ff3bf82a epochNumber=0
{"provingJobId":"1:BASE_PARITY:9c05b785ff3bf82a"}
15:15:07 [15:15:07.641]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.643]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.644]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ac80085ca76d4d5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5ac80085ca76d4d5"}
15:15:07 [15:15:07.644]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testbUdMCq/2 with map size 1073741824
15:15:07 [15:15:07.647]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.665]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a05a50fc8d09a9ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:a05a50fc8d09a9ff"}
15:15:07 [15:15:07.666]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbUdMCq/1 with map size 1073741824
15:15:07 [15:15:07.667]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.692]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da8494122f03c54e epochNumber=1
{"provingJobId":"1:BASE_PARITY:da8494122f03c54e"}
15:15:07 [15:15:07.694]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9lFdOw/1 with map size 1073741824
15:15:07 [15:15:07.698]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.722]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.723]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f52d63b3b1b7f602 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f52d63b3b1b7f602"}
15:15:07 [15:15:07.724]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAU4DfQ/1 with map size 1073741824
15:15:07 [15:15:07.726]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.745]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:243330ac35cb2276 epochNumber=1
{"provingJobId":"1:BASE_PARITY:243330ac35cb2276"}
15:15:07 [15:15:07.752]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e8a1d248596a8bc epochNumber=2
{"provingJobId":"1:BASE_PARITY:9e8a1d248596a8bc"}
15:15:07 [15:15:07.753]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAU4DfQ/2 with map size 1073741824
15:15:07 [15:15:07.755]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7d44730415cbe442 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7d44730415cbe442"}
15:15:07 [15:15:07.787]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.788]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42a3f289639cfcf0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:42a3f289639cfcf0"}
15:15:07 [15:15:07.788]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmAeLdE/1 with map size 1073741824
15:15:07 [15:15:07.790]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7566b2950d23225d epochNumber=1
{"provingJobId":"1:BASE_PARITY:7566b2950d23225d"}
15:15:07 [15:15:07.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b40f52d5b74f6032 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b40f52d5b74f6032"}
15:15:07 [15:15:07.814]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testmAeLdE/2 with map size 1073741824
15:15:07 [15:15:07.816]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.835]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31a7be5c45e05cc7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:31a7be5c45e05cc7"}
15:15:07 [15:15:07.844]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.844]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eeab6e17cd79d72e epochNumber=1
{"provingJobId":"1:BASE_PARITY:eeab6e17cd79d72e"}
15:15:07 [15:15:07.845]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testz6UoEv/1 with map size 1073741824
15:15:07 [15:15:07.847]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.863]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7410b1977c869aab epochNumber=1
{"provingJobId":"1:BASE_PARITY:7410b1977c869aab"}
15:15:07 [15:15:07.868]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92779ddfc20a90f4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:92779ddfc20a90f4"}
15:15:07 [15:15:07.869]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testz6UoEv/2 with map size 1073741824
15:15:07 [15:15:07.871]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.888]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d9ca82419b849404 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d9ca82419b849404"}
15:15:07 [15:15:07.896]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.897]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d76ad6ce4bd44ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d76ad6ce4bd44ba"}
15:15:07 [15:15:07.900]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDY9PZg/1 with map size 1073741824
15:15:07 [15:15:07.902]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.919]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3883ebcde2e0c72 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e3883ebcde2e0c72"}
15:15:07 [15:15:07.924]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3d76ad6ce4bd44ba type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3d76ad6ce4bd44ba"}
15:15:07 [15:15:07.933]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.933]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d807f83de9eb546d epochNumber=1
{"provingJobId":"1:BASE_PARITY:d807f83de9eb546d"}
15:15:07 [15:15:07.934]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testx15onc/1 with map size 1073741824
15:15:07 [15:15:07.935]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.951]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96aad1c7ffa06b46 epochNumber=1
{"provingJobId":"1:BASE_PARITY:96aad1c7ffa06b46"}
15:15:07 [15:15:07.957]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d807f83de9eb546d type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:d807f83de9eb546d"}
15:15:07 [15:15:07.966]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.966]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:023106938be60456 epochNumber=1
{"provingJobId":"1:BASE_PARITY:023106938be60456"}
15:15:07 [15:15:07.967]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfhipJZ/1 with map size 1073741824
15:15:07 [15:15:07.968]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:07 [15:15:07.984]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f313116bc60c874e epochNumber=1
{"provingJobId":"1:BASE_PARITY:f313116bc60c874e"}
15:15:07 [15:15:07.989]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:023106938be60456 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:023106938be60456"}
15:15:07 [15:15:07.992]
INFO:
prover-client:proving-broker Proving Broker started
15:15:07 [15:15:07.993]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c26e113f1110ff73 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c26e113f1110ff73"}
15:15:07 [15:15:07.995]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZT8kRB/1 with map size 1073741824
15:15:07 [15:15:07.998]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:08 [15:15:08.015]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c26e113f1110ff73
{"provingJobId":"1:BASE_PARITY:c26e113f1110ff73"}
15:15:08 [15:15:08.015]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c26e113f1110ff73 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c26e113f1110ff73"}
15:15:08 [15:15:08.021]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99d48c1643ead582 epochNumber=1
{"provingJobId":"1:BASE_PARITY:99d48c1643ead582"}
15:15:08 [15:15:08.029]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c26e113f1110ff73 has already been completed
{"provingJobId":"1:BASE_PARITY:c26e113f1110ff73"}
15:15:08 [15:15:08.033]
INFO:
prover-client:proving-broker Proving Broker started
15:15:08 [15:15:08.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c87608744f4995d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c87608744f4995d3"}
15:15:08 [15:15:08.034]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5uilm2/1 with map size 1073741824
15:15:08 [15:15:08.036]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:08 [15:15:08.052]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1e4795854f163bb epochNumber=2
{"provingJobId":"1:BASE_PARITY:d1e4795854f163bb"}
15:15:08 [15:15:08.053]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5uilm2/2 with map size 1073741824
15:15:08 [15:15:08.055]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:09 [15:15:09.123]
INFO:
prover-client:proving-broker Proving Broker started
15:15:09 [15:15:09.130]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c87608744f4995d3 settled=false
{"provingJobId":"1:BASE_PARITY:c87608744f4995d3","status":"pending"}
15:15:09 [15:15:09.132]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d1e4795854f163bb settled=false
{"provingJobId":"1:BASE_PARITY:d1e4795854f163bb","status":"pending"}
15:15:09 [15:15:09.133]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c87608744f4995d3 type=BASE_PARITY startedAt=1750086908071 older agent has taken job
{"provingJobId":"1:BASE_PARITY:c87608744f4995d3"}
15:15:09 [15:15:09.133]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c87608744f4995d3 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:c87608744f4995d3"}
15:15:09 [15:15:09.137]
INFO:
prover-client:proving-broker Proving Broker started
15:15:09 [15:15:09.137]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc2572c6580947b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bc2572c6580947b1"}
15:15:09 [15:15:09.138]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeOfL7T/1 with map size 1073741824
15:15:09 [15:15:09.140]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:09 [15:15:09.154]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94f746e577507f0d epochNumber=2
{"provingJobId":"1:BASE_PARITY:94f746e577507f0d"}
15:15:09 [15:15:09.155]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testeOfL7T/2 with map size 1073741824
15:15:09 [15:15:09.157]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:10 [15:15:10.174]
INFO:
prover-client:proving-broker Proving Broker started
15:15:10 [15:15:10.176]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bc2572c6580947b1 settled=false
{"provingJobId":"1:BASE_PARITY:bc2572c6580947b1","status":"pending"}
15:15:10 [15:15:10.178]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:94f746e577507f0d settled=false
{"provingJobId":"1:BASE_PARITY:94f746e577507f0d","status":"pending"}
15:15:10 [15:15:10.179]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bc2572c6580947b1 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:bc2572c6580947b1"}
15:15:10 [15:15:10.183]
INFO:
prover-client:proving-broker Proving Broker started
15:15:10 [15:15:10.183]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de0e0ccf67b8e143 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de0e0ccf67b8e143"}
15:15:10 [15:15:10.185]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDyzzRa/1 with map size 1073741824
15:15:10 [15:15:10.189]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:10 [15:15:10.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55448b955e094d4a epochNumber=2
{"provingJobId":"1:BASE_PARITY:55448b955e094d4a"}
15:15:10 [15:15:10.209]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDyzzRa/2 with map size 1073741824
15:15:10 [15:15:10.211]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:11 [15:15:11.228]
INFO:
prover-client:proving-broker Proving Broker started
15:15:11 [15:15:11.230]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:de0e0ccf67b8e143 settled=false
{"provingJobId":"1:BASE_PARITY:de0e0ccf67b8e143","status":"pending"}
15:15:11 [15:15:11.231]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:55448b955e094d4a settled=false
{"provingJobId":"1:BASE_PARITY:55448b955e094d4a","status":"pending"}
15:15:11 [15:15:11.232]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:de0e0ccf67b8e143 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:de0e0ccf67b8e143"}
15:15:11 [15:15:11.232]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:de0e0ccf67b8e143 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:de0e0ccf67b8e143"}
15:15:11 [15:15:11.242]
INFO:
prover-client:proving-broker Proving Broker started
15:15:11 [15:15:11.242]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:467d6527795d829e epochNumber=1
{"provingJobId":"1:BASE_PARITY:467d6527795d829e"}
15:15:11 [15:15:11.243]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testY9kHtz/1 with map size 1073741824
15:15:11 [15:15:11.246]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:11 [15:15:11.264]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:327b84fdd4a39ec6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:327b84fdd4a39ec6"}
15:15:11 [15:15:11.264]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testY9kHtz/2 with map size 1073741824
15:15:11 [15:15:11.266]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:11 [15:15:11.288]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:467d6527795d829e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:467d6527795d829e"}
15:15:11 [15:15:11.297]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:327b84fdd4a39ec6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:327b84fdd4a39ec6"}
15:15:11 [15:15:11.310]
INFO:
prover-client:proving-broker Proving Broker started
15:15:11 [15:15:11.311]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d706f9ed32c7769c epochNumber=1
{"provingJobId":"1:BASE_PARITY:d706f9ed32c7769c"}
15:15:11 [15:15:11.311]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRTqkT6/1 with map size 1073741824
15:15:11 [15:15:11.315]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:11 [15:15:11.337]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88f440e80e775362 epochNumber=2
{"provingJobId":"1:BASE_PARITY:88f440e80e775362"}
15:15:11 [15:15:11.338]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testRTqkT6/2 with map size 1073741824
15:15:11 [15:15:11.341]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:11 [15:15:11.363]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d706f9ed32c7769c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d706f9ed32c7769c"}
15:15:11 [15:15:11.363]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d706f9ed32c7769c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d706f9ed32c7769c"}
15:15:11 [15:15:11.371]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:88f440e80e775362 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:88f440e80e775362"}
15:15:11 [15:15:11.383]
INFO:
prover-client:proving-broker Proving Broker started
15:15:11 [15:15:11.385]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:cc464306d5648235 err=test error
{"provingJoId":"1:BASE_PARITY:cc464306d5648235"}
15:15:11 [15:15:11.388]
INFO:
prover-client:proving-broker Proving Broker started
15:15:11 [15:15:11.389]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:14c6e5c27ee76ba9 not found
{"provingJobId":"1:BASE_PARITY:14c6e5c27ee76ba9"}
15:15:11 [15:15:11.395]
INFO:
prover-client:proving-broker Proving Broker started
15:15:11 [15:15:11.396]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c394221ca38c263 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c394221ca38c263"}
15:15:11 [15:15:11.398]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZC0SQv/1 with map size 1073741824
15:15:11 [15:15:11.404]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:11 [15:15:11.428]
INFO:
prover-client:proving-broker Proving Broker started
15:15:11 [15:15:11.429]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ca91003a5c5e022 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ca91003a5c5e022"}
15:15:11 [15:15:11.429]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrsgmOA/1 with map size 1073741824
15:15:11 [15:15:11.432]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:11 [15:15:11.554]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9ca91003a5c5e022 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9ca91003a5c5e022"}
15:15:11 [15:15:11.560]
INFO:
prover-client:proving-broker Proving Broker started
15:15:11 [15:15:11.561]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cef1464da8234640 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cef1464da8234640"}
15:15:11 [15:15:11.562]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testl8sf8i/1 with map size 1073741824
15:15:11 [15:15:11.563]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:11 [15:15:11.688]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cef1464da8234640 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:cef1464da8234640"}
15:15:11 [15:15:11.702]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57fc5c14bdca2e3d epochNumber=10
{"provingJobId":"1:BASE_PARITY:57fc5c14bdca2e3d"}
15:15:11 [15:15:11.704]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testl8sf8i/10 with map size 1073741824
15:15:11 [15:15:11.708]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:11 [15:15:11.740]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:11 [15:15:11.740]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:15:11 [15:15:11.858]
INFO:
prover-client:proving-broker Proving Broker started
15:15:11 [15:15:11.859]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:924871c9399efa55 epochNumber=1
{"provingJobId":"1:BASE_PARITY:924871c9399efa55"}
15:15:11 [15:15:11.860]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsylYin/1 with map size 1073741824
15:15:11 [15:15:11.861]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.059]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:924871c9399efa55 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:924871c9399efa55"}
15:15:12 [15:15:12.103]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.104]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:825f072af25530e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:825f072af25530e6"}
15:15:12 [15:15:12.105]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testT2uHyd/1 with map size 1073741824
15:15:12 [15:15:12.107]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.129]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:825f072af25530e6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:825f072af25530e6"}
15:15:12 [15:15:12.132]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8050bfcde923b4d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8050bfcde923b4d9"}
15:15:12 [15:15:12.135]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHY5gKO/1 with map size 1073741824
15:15:12 [15:15:12.136]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.157]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8050bfcde923b4d9 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:8050bfcde923b4d9"}
15:15:12 [15:15:12.158]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8050bfcde923b4d9 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:8050bfcde923b4d9"}
15:15:12 [15:15:12.169]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.170]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2b6a9d48f041c4d epochNumber=1
{"provingJobId":"1:BASE_PARITY:e2b6a9d48f041c4d"}
15:15:12 [15:15:12.171]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCK2rEE/1 with map size 1073741824
15:15:12 [15:15:12.172]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.196]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e2b6a9d48f041c4d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e2b6a9d48f041c4d"}
15:15:12 [15:15:12.212]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.213]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da5cb256e3c5ad06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:da5cb256e3c5ad06"}
15:15:12 [15:15:12.214]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4KNfWW/1 with map size 1073741824
15:15:12 [15:15:12.216]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.235]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:da5cb256e3c5ad06 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:da5cb256e3c5ad06"}
15:15:12 [15:15:12.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6867fc1abdb6636 epochNumber=3
{"provingJobId":"1:BASE_PARITY:a6867fc1abdb6636"}
15:15:12 [15:15:12.238]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test4KNfWW/3 with map size 1073741824
15:15:12 [15:15:12.241]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.263]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:12 [15:15:12.263]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:15:12 [15:15:12.283]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:da5cb256e3c5ad06 err=test error
{"provingJoId":"1:BASE_PARITY:da5cb256e3c5ad06"}
15:15:12 [15:15:12.302]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testK6QvOM/1 with map size 1073741824
15:15:12 [15:15:12.305]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.332]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testK6QvOM/2 with map size 1073741824
15:15:12 [15:15:12.334]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.358]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.360]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4268c1cc3dac28cb settled=false
{"provingJobId":"1:BASE_PARITY:4268c1cc3dac28cb","status":"pending"}
15:15:12 [15:15:12.361]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4af0826ed4544574 settled=false
{"provingJobId":"1:BASE_PARITY:4af0826ed4544574","status":"pending"}
15:15:12 [15:15:12.370]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWS3VHJ/1 with map size 1073741824
15:15:12 [15:15:12.375]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.414]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWS3VHJ/2 with map size 1073741824
15:15:12 [15:15:12.419]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.458]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.461]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:aa21d3b8153b3733 settled=true
{"provingJobId":"1:BASE_PARITY:aa21d3b8153b3733","status":"fulfilled"}
15:15:12 [15:15:12.463]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:c62f747ea4ab43f3 settled=true
{"provingJobId":"2:BASE_PARITY:c62f747ea4ab43f3","status":"fulfilled"}
15:15:12 [15:15:12.473]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTKRpur/1 with map size 1073741824
15:15:12 [15:15:12.477]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.523]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTKRpur/2 with map size 1073741824
15:15:12 [15:15:12.528]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.558]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.560]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:89ea60564a15ed33 settled=true
{"provingJobId":"1:BASE_PARITY:89ea60564a15ed33","status":"fulfilled"}
15:15:12 [15:15:12.563]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:461a2ac8f3ed4209 settled=false
{"provingJobId":"1:BASE_PARITY:461a2ac8f3ed4209","status":"pending"}
15:15:12 [15:15:12.569]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.570]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61fbf4126815cf46 epochNumber=1
{"provingJobId":"1:BASE_PARITY:61fbf4126815cf46"}
15:15:12 [15:15:12.574]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7p7lM0/1 with map size 1073741824
15:15:12 [15:15:12.579]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.619]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.620]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4f90bfd0cdb66b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4f90bfd0cdb66b5"}
15:15:12 [15:15:12.621]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:c4f90bfd0cdb66b5: Error: db error: Error: db error
15:15:12 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:c4f90bfd0cdb66b5"}
15:15:12 [15:15:12.627]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.627]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4dd1c7041a101001 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4dd1c7041a101001"}
15:15:12 [15:15:12.629]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testw7pwDa/1 with map size 1073741824
15:15:12 [15:15:12.633]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.654]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4dd1c7041a101001 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4dd1c7041a101001"}
15:15:12 [15:15:12.654]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4dd1c7041a101001 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4dd1c7041a101001"}
15:15:12 [15:15:12.667]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.667]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ea25a06b19efd0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ea25a06b19efd0e"}
15:15:12 [15:15:12.669]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTMh6Tj/1 with map size 1073741824
15:15:12 [15:15:12.671]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.694]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7ea25a06b19efd0e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7ea25a06b19efd0e"}
15:15:12 [15:15:12.696]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7ea25a06b19efd0e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7ea25a06b19efd0e"}
15:15:12 [15:15:12.697]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:7ea25a06b19efd0e: Error: db error
15:15:12 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:7ea25a06b19efd0e"}
15:15:12 [15:15:12.705]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.705]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48baacd0ebf048ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:48baacd0ebf048ca"}
15:15:12 [15:15:12.706]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiyRo4s/1 with map size 1073741824
15:15:12 [15:15:12.708]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.730]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:48baacd0ebf048ca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:48baacd0ebf048ca"}
15:15:12 [15:15:12.730]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:48baacd0ebf048ca type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:48baacd0ebf048ca"}
15:15:12 [15:15:12.742]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:669992892e214e5a epochNumber=1
{"provingJobId":"1:BASE_PARITY:669992892e214e5a"}
15:15:12 [15:15:12.743]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBBruSC/1 with map size 1073741824
15:15:12 [15:15:12.747]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.769]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:669992892e214e5a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:669992892e214e5a"}
15:15:12 [15:15:12.770]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:669992892e214e5a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:669992892e214e5a"}
15:15:12 [15:15:12.771]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:669992892e214e5a jobErr=test error: Error: db error
15:15:12 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:669992892e214e5a"}
15:15:12 [15:15:12.779]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.780]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:98af1cf480989da8 not found
{"provingJobId":"1:BASE_PARITY:98af1cf480989da8"}
15:15:12 [15:15:12.795]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.797]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f75c5c7e4e113d9e err=test error
{"provingJoId":"1:BASE_PARITY:f75c5c7e4e113d9e"}
15:15:12 [15:15:12.806]
INFO:
prover-client:proving-broker Proving Broker started
15:15:12 [15:15:12.818]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed8ab86bf57051e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed8ab86bf57051e1"}
15:15:12 [15:15:12.819]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbYoTr8/1 with map size 1073741824
15:15:12 [15:15:12.822]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.843]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ed8ab86bf57051e1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ed8ab86bf57051e1"}
15:15:12 [15:15:12.844]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ed8ab86bf57051e1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ed8ab86bf57051e1"}
15:15:12 [15:15:12.863]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:3218fb1125f8e29a epochNumber=2
{"provingJobId":"2:BASE_PARITY:3218fb1125f8e29a"}
15:15:12 [15:15:12.864]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testbYoTr8/2 with map size 1073741824
15:15:12 [15:15:12.867]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:12 [15:15:12.898]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:3218fb1125f8e29a type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:3218fb1125f8e29a"}
15:15:12 [15:15:12.899]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:3218fb1125f8e29a type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:3218fb1125f8e29a"}
15:15:12 [15:15:12.969]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:fdf605b7b3035016 epochNumber=3
{"provingJobId":"3:BASE_PARITY:fdf605b7b3035016"}
15:15:12 [15:15:12.971]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testbYoTr8/3 with map size 1073741824
15:15:12 [15:15:12.976]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:13 [15:15:13.015]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:13 [15:15:13.015]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:15:13 [15:15:13.069]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:b1268e5cc99b2052 epochNumber=4
{"provingJobId":"4:BASE_PARITY:b1268e5cc99b2052"}
15:15:13 [15:15:13.070]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testbYoTr8/4 with map size 1073741824
15:15:13 [15:15:13.072]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:13 [15:15:13.122]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:13 [15:15:13.123]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
15:15:13 [15:15:13.158]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:46ca2d2cb0a33d09 epochNumber=5
{"provingJobId":"5:BASE_PARITY:46ca2d2cb0a33d09"}
15:15:13 [15:15:13.159]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testbYoTr8/5 with map size 1073741824
15:15:13 [15:15:13.164]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:15:13 [15:15:13.201]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:15:13 [15:15:13.202]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
15:15:13
PASS src/proving_broker/proving_broker.test.ts (
13.728 s)
15:15:13 ProvingBroker
15:15:13 Producer API
15:15:13
✓ refuses stale jobs (55 ms)
15:15:13
✓ enqueues jobs (4 ms)
15:15:13
✓ ignores duplicate jobs (5 ms)
15:15:13
✓ reports correct status when enqueuing repeat jobs (4 ms)
15:15:13
✓ throws an error in case of duplicate job IDs (23 ms)
15:15:13
✓ returns not-found status for non-existing jobs (2 ms)
15:15:13
✓ cancels jobs in queue (2 ms)
15:15:13
✓ cancels jobs in-progress (2 ms)
15:15:13
✓ returns job result if successful (3 ms)
15:15:13
✓ returns job error if failed (12 ms)
15:15:13
✓ correctly returns job status for concurrent writes (2 ms)
15:15:13
✓ refuses stale jobs (31 ms)
15:15:13
✓ enqueues jobs (44 ms)
15:15:13
✓ ignores duplicate jobs (31 ms)
15:15:13
✓ reports correct status when enqueuing repeat jobs (29 ms)
15:15:13
✓ throws an error in case of duplicate job IDs (23 ms)
15:15:13
✓ returns not-found status for non-existing jobs (2 ms)
15:15:13
✓ cancels jobs in queue (41 ms)
15:15:13
✓ cancels jobs in-progress (27 ms)
15:15:13
✓ returns job result if successful (29 ms)
15:15:13
✓ returns job error if failed (32 ms)
15:15:13
✓ correctly returns job status for concurrent writes (36 ms)
15:15:13 Consumer API
15:15:13
✓ returns undefined if no jobs are available (2 ms)
15:15:13
✓ returns jobs in priority order (2 ms)
15:15:13
✓ returns undefined if no jobs are available for the given allowList (1 ms)
15:15:13
✓ returns a job if it is in the allowList (3 ms)
15:15:13
✓ returns the most important job if it is in the allowList (2 ms)
15:15:13
✓ returns any job if filter is empty (2 ms)
15:15:13
✓ returns a new job when reporting job success (3 ms)
15:15:13
✓ returns a new job when reporting permanent error (2 ms)
15:15:13
✓ returns a new job when reporting retry-able error (2 ms)
15:15:13
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
15:15:13
✓ returns a new job if job is already in progress elsewhere (1055 ms)
15:15:13
✓ avoids sending the same job to a new agent after a restart (1005 ms)
15:15:13
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
15:15:13
✓ tracks job result if in progress (3 ms)
15:15:13
✓ tracks job result even if job is in queue (3 ms)
15:15:13
✓ ignores reported job error if unknown job (1 ms)
15:15:13
✓ ignores job result if unknown job (1 ms)
15:15:13
✓ returns undefined if no jobs are available (2 ms)
15:15:13
✓ returns jobs in priority order (49 ms)
15:15:13
✓ returns undefined if no jobs are available for the given allowList (28 ms)
15:15:13
✓ returns a job if it is in the allowList (66 ms)
15:15:13
✓ returns the most important job if it is in the allowList (57 ms)
15:15:13
✓ returns any job if filter is empty (53 ms)
15:15:13
✓ returns a new job when reporting job success (35 ms)
15:15:13
✓ returns a new job when reporting permanent error (33 ms)
15:15:13
✓ returns a new job when reporting retry-able error (26 ms)
15:15:13
✓ returns a new job when reporting progress if current one is cancelled (40 ms)
15:15:13
✓ returns a new job if job is already in progress elsewhere (1104 ms)
15:15:13
✓ avoids sending the same job to a new agent after a restart (1046 ms)
15:15:13
✓ avoids sending a completed job to a new agent after a restart (1058 ms)
15:15:13
✓ tracks job result if in progress (68 ms)
15:15:13
✓ tracks job result even if job is in queue (71 ms)
15:15:13
✓ ignores reported job error if unknown job (6 ms)
15:15:13
✓ ignores job result if unknown job (5 ms)
15:15:13 Timeouts
15:15:13
✓ tracks in progress jobs (2 ms)
15:15:13
✓ re-enqueues jobs that time out (109 ms)
15:15:13
✓ cancel stale jobs that time out (236 ms)
15:15:13
✓ keeps the jobs in progress while it is alive (203 ms)
15:15:13
✓ tracks in progress jobs (34 ms)
15:15:13
✓ re-enqueues jobs that time out (132 ms)
15:15:13
✓ cancel stale jobs that time out (296 ms)
15:15:13
✓ keeps the jobs in progress while it is alive (243 ms)
15:15:13 Retries
15:15:13
✓ retries jobs (3 ms)
15:15:13
✓ retries up to a maximum number of times (2 ms)
15:15:13
✓ passing retry=false does not retry (2 ms)
15:15:13
✓ does not retry if job is stale (28 ms)
15:15:13
✓ retries jobs (32 ms)
15:15:13
✓ retries up to a maximum number of times (35 ms)
15:15:13
✓ passing retry=false does not retry (41 ms)
15:15:13
✓ does not retry if job is stale (81 ms)
15:15:13 Database management
15:15:13
✓ re-enqueues proof requests on start (2 ms)
15:15:13
✓ restores proof results on start (1 ms)
15:15:13
✓ only re-enqueues unfinished jobs (1 ms)
15:15:13
✓ saves job when enqueued (3 ms)
15:15:13
✓ does not retain job if database fails to save (2 ms)
15:15:13
✓ saves job result (1 ms)
15:15:13
✓ saves result even if database fails to save (2 ms)
15:15:13
✓ saves job error (2 ms)
15:15:13
✓ saves job error even if database fails to save (2 ms)
15:15:13
✓ does not save job result if job is unknown (1 ms)
15:15:13
✓ does not save job error if job is unknown (1 ms)
15:15:13
✓ cleans up old jobs periodically (259 ms)
15:15:13
✓ re-enqueues proof requests on start (73 ms)
15:15:13
✓ restores proof results on start (100 ms)
15:15:13
✓ only re-enqueues unfinished jobs (101 ms)
15:15:13
✓ saves job when enqueued (46 ms)
15:15:13
✓ does not retain job if database fails to save (8 ms)
15:15:13
✓ saves job result (40 ms)
15:15:13
✓ saves result even if database fails to save (37 ms)
15:15:13
✓ saves job error (37 ms)
15:15:13
✓ saves job error even if database fails to save (34 ms)
15:15:13
✓ does not save job result if job is unknown (8 ms)
15:15:13
✓ does not save job error if job is unknown (13 ms)
15:15:13
✓ cleans up old jobs periodically (433 ms)
15:15:13
15:15:13
Test Suites: 1 passed, 1 total
15:15:13
Tests: 96 passed, 96 total
15:15:13
Snapshots: 0 total
15:15:13
Time: 13.846 s
15:15:13
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
15:15:13
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?