Parent Log:
http://ci.aztec-labs.com/4b746e1186bf09cc
Command: 0e6cfc077fbe1818 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/f17855f94a58189a98dd378743e67b774ea38144
Env: REF_NAME=gh-readonly-queue/next/pr-15066-47bd93499e6606131b6ececa3536632cf5e84409 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Wed Jun 18 10:00:42 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15066_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
10:00:46 [10:00:46.401]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.408]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0875dc41279689a0 epochNumber=42
{"provingJobId":"1:BASE_PARITY:0875dc41279689a0"}
10:00:46 [10:00:46.412]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:03ab4be8dd433c70 epochNumber=1
{"provingJobId":"1:BASE_PARITY:03ab4be8dd433c70"}
10:00:46 [10:00:46.439]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.440]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f65bbb28820af63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f65bbb28820af63"}
10:00:46 [10:00:46.441]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31d0528ba025ac09 epochNumber=1
{"provingJobId":"1:BASE_PARITY:31d0528ba025ac09"}
10:00:46 [10:00:46.443]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.444]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b3c9a335c817d5c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b3c9a335c817d5c0"}
10:00:46 [10:00:46.445]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b3c9a335c817d5c0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b3c9a335c817d5c0"}
10:00:46 [10:00:46.447]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.448]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a048a78ed707ab8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6a048a78ed707ab8"}
10:00:46 [10:00:46.449]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6a048a78ed707ab8 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6a048a78ed707ab8"}
10:00:46 [10:00:46.450]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6a048a78ed707ab8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6a048a78ed707ab8"}
10:00:46 [10:00:46.451]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6a048a78ed707ab8 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6a048a78ed707ab8"}
10:00:46 [10:00:46.453]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:503a166852cdab87 epochNumber=1
{"provingJobId":"1:BASE_PARITY:503a166852cdab87"}
10:00:46 [10:00:46.480]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.482]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff41f8dde1c02026 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff41f8dde1c02026"}
10:00:46 [10:00:46.483]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ff41f8dde1c02026
{"provingJobId":"1:BASE_PARITY:ff41f8dde1c02026"}
10:00:46 [10:00:46.484]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ff41f8dde1c02026 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ff41f8dde1c02026"}
10:00:46 [10:00:46.484]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ff41f8dde1c02026 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ff41f8dde1c02026"}
10:00:46 [10:00:46.485]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.486]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c1c51090836dd01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7c1c51090836dd01"}
10:00:46 [10:00:46.487]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7c1c51090836dd01
{"provingJobId":"1:BASE_PARITY:7c1c51090836dd01"}
10:00:46 [10:00:46.487]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7c1c51090836dd01 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7c1c51090836dd01"}
10:00:46 [10:00:46.488]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.489]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea26ac7d8d1aeba8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea26ac7d8d1aeba8"}
10:00:46 [10:00:46.490]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ea26ac7d8d1aeba8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ea26ac7d8d1aeba8"}
10:00:46 [10:00:46.490]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ea26ac7d8d1aeba8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ea26ac7d8d1aeba8"}
10:00:46 [10:00:46.491]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.492]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc94c7c069437b8f epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc94c7c069437b8f"}
10:00:46 [10:00:46.492]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fc94c7c069437b8f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fc94c7c069437b8f"}
10:00:46 [10:00:46.493]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fc94c7c069437b8f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fc94c7c069437b8f"}
10:00:46 [10:00:46.494]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.494]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52e74e4223b0dbbe epochNumber=0
{"provingJobId":"1:BASE_PARITY:52e74e4223b0dbbe"}
10:00:46 [10:00:46.495]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:52e74e4223b0dbbe epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:52e74e4223b0dbbe"}
10:00:46 [10:00:46.495]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a5585dcc320164c4 epochNumber=0
{"provingJobId":"1:BASE_PARITY:a5585dcc320164c4"}
10:00:46 [10:00:46.495]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:52e74e4223b0dbbe epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:52e74e4223b0dbbe"}
10:00:46 [10:00:46.495]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ab3968eccb759c0 epochNumber=0
{"provingJobId":"1:BASE_PARITY:2ab3968eccb759c0"}
10:00:46 [10:00:46.503]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.505]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfd2982fb4a01822 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dfd2982fb4a01822"}
10:00:46 [10:00:46.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43c5c090c0079291 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43c5c090c0079291"}
10:00:46 [10:00:46.508]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.508]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1213db4ff75d530 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c1213db4ff75d530"}
10:00:46 [10:00:46.509]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:109fea34f11d7ff7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:109fea34f11d7ff7"}
10:00:46 [10:00:46.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78a34e42cce65a21 epochNumber=1
{"provingJobId":"1:BASE_PARITY:78a34e42cce65a21"}
10:00:46 [10:00:46.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6fc81fa68056e652 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6fc81fa68056e652"}
10:00:46 [10:00:46.511]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25575806ee9e27e4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:25575806ee9e27e4"}
10:00:46 [10:00:46.512]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20b53ebdb7f851df epochNumber=1
{"provingJobId":"1:BASE_PARITY:20b53ebdb7f851df"}
10:00:46 [10:00:46.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc707f243390c72f epochNumber=1
{"provingJobId":"1:BASE_PARITY:bc707f243390c72f"}
10:00:46 [10:00:46.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5128d7ce351139b epochNumber=2
{"provingJobId":"1:BASE_PARITY:c5128d7ce351139b"}
10:00:46 [10:00:46.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca7bbbd473d62d30 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca7bbbd473d62d30"}
10:00:46 [10:00:46.514]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.515]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:038c9c68f734ae1e epochNumber=1
{"provingJobId":"1:BASE_PARITY:038c9c68f734ae1e"}
10:00:46 [10:00:46.515]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:459246efee493adb epochNumber=1
{"provingJobId":"1:BASE_PARITY:459246efee493adb"}
10:00:46 [10:00:46.515]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:200c6d5ee4347352 epochNumber=2
{"provingJobId":"1:BASE_PARITY:200c6d5ee4347352"}
10:00:46 [10:00:46.516]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:126e3de3d38e0ab3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:126e3de3d38e0ab3"}
10:00:46 [10:00:46.517]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.517]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2eb2813b088b75c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2eb2813b088b75c7"}
10:00:46 [10:00:46.518]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1bac258e16ddc67b epochNumber=1
{"provingJobId":"1:BASE_PARITY:1bac258e16ddc67b"}
10:00:46 [10:00:46.518]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2eb2813b088b75c7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2eb2813b088b75c7"}
10:00:46 [10:00:46.519]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61d2e24f5a145fe9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:61d2e24f5a145fe9"}
10:00:46 [10:00:46.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79a06e8a1a6fa219 epochNumber=1
{"provingJobId":"1:BASE_PARITY:79a06e8a1a6fa219"}
10:00:46 [10:00:46.520]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:61d2e24f5a145fe9 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:61d2e24f5a145fe9"}
10:00:46 [10:00:46.521]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.522]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b96acd3e3f6453e epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b96acd3e3f6453e"}
10:00:46 [10:00:46.522]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c53bcd49065d4f90 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c53bcd49065d4f90"}
10:00:46 [10:00:46.522]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1b96acd3e3f6453e type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:1b96acd3e3f6453e"}
10:00:46 [10:00:46.523]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.524]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf61e3e5d55ae8b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf61e3e5d55ae8b9"}
10:00:46 [10:00:46.524]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:bf61e3e5d55ae8b9
{"provingJobId":"1:BASE_PARITY:bf61e3e5d55ae8b9"}
10:00:46 [10:00:46.524]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bf61e3e5d55ae8b9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:bf61e3e5d55ae8b9"}
10:00:46 [10:00:46.525]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61759f019a477cd9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:61759f019a477cd9"}
10:00:46 [10:00:46.525]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bf61e3e5d55ae8b9 has already been completed
{"provingJobId":"1:BASE_PARITY:bf61e3e5d55ae8b9"}
10:00:46 [10:00:46.526]
INFO:
prover-client:proving-broker Proving Broker started
10:00:46 [10:00:46.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a68b3030eb15f7c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a68b3030eb15f7c6"}
10:00:46 [10:00:46.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ac82686ff072d80 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2ac82686ff072d80"}
10:00:47 [10:00:47.579]
INFO:
prover-client:proving-broker Proving Broker started
10:00:47 [10:00:47.580]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a68b3030eb15f7c6 settled=false
{"provingJobId":"1:BASE_PARITY:a68b3030eb15f7c6","status":"pending"}
10:00:47 [10:00:47.580]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2ac82686ff072d80 settled=false
{"provingJobId":"1:BASE_PARITY:2ac82686ff072d80","status":"pending"}
10:00:47 [10:00:47.581]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a68b3030eb15f7c6 type=BASE_PARITY startedAt=1750240846527 older agent has taken job
{"provingJobId":"1:BASE_PARITY:a68b3030eb15f7c6"}
10:00:47 [10:00:47.582]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a68b3030eb15f7c6 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:a68b3030eb15f7c6"}
10:00:47 [10:00:47.586]
INFO:
prover-client:proving-broker Proving Broker started
10:00:47 [10:00:47.587]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3c04556f464ff97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3c04556f464ff97"}
10:00:47 [10:00:47.588]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e0dae4bf4dbcdfa epochNumber=2
{"provingJobId":"1:BASE_PARITY:3e0dae4bf4dbcdfa"}
10:00:48 [10:00:48.589]
INFO:
prover-client:proving-broker Proving Broker started
10:00:48 [10:00:48.589]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a3c04556f464ff97 settled=false
{"provingJobId":"1:BASE_PARITY:a3c04556f464ff97","status":"pending"}
10:00:48 [10:00:48.590]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3e0dae4bf4dbcdfa settled=false
{"provingJobId":"1:BASE_PARITY:3e0dae4bf4dbcdfa","status":"pending"}
10:00:48 [10:00:48.590]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a3c04556f464ff97 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:a3c04556f464ff97"}
10:00:48 [10:00:48.592]
INFO:
prover-client:proving-broker Proving Broker started
10:00:48 [10:00:48.592]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:500a2dc39a9e085b epochNumber=1
{"provingJobId":"1:BASE_PARITY:500a2dc39a9e085b"}
10:00:48 [10:00:48.593]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3b4040c716f09ce epochNumber=2
{"provingJobId":"1:BASE_PARITY:d3b4040c716f09ce"}
10:00:49 [10:00:49.594]
INFO:
prover-client:proving-broker Proving Broker started
10:00:49 [10:00:49.595]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:500a2dc39a9e085b settled=false
{"provingJobId":"1:BASE_PARITY:500a2dc39a9e085b","status":"pending"}
10:00:49 [10:00:49.595]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d3b4040c716f09ce settled=false
{"provingJobId":"1:BASE_PARITY:d3b4040c716f09ce","status":"pending"}
10:00:49 [10:00:49.595]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:500a2dc39a9e085b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:500a2dc39a9e085b"}
10:00:49 [10:00:49.596]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:500a2dc39a9e085b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:500a2dc39a9e085b"}
10:00:49 [10:00:49.597]
INFO:
prover-client:proving-broker Proving Broker started
10:00:49 [10:00:49.598]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3240d33ad9ae4f93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3240d33ad9ae4f93"}
10:00:49 [10:00:49.598]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:831e3855da6b3866 epochNumber=2
{"provingJobId":"1:BASE_PARITY:831e3855da6b3866"}
10:00:49 [10:00:49.598]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3240d33ad9ae4f93 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3240d33ad9ae4f93"}
10:00:49 [10:00:49.599]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:831e3855da6b3866 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:831e3855da6b3866"}
10:00:49 [10:00:49.599]
INFO:
prover-client:proving-broker Proving Broker started
10:00:49 [10:00:49.600]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9542d0cfc2ebc30 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9542d0cfc2ebc30"}
10:00:49 [10:00:49.600]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ee70ff5faa429c5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2ee70ff5faa429c5"}
10:00:49 [10:00:49.600]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b9542d0cfc2ebc30 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b9542d0cfc2ebc30"}
10:00:49 [10:00:49.601]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b9542d0cfc2ebc30 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b9542d0cfc2ebc30"}
10:00:49 [10:00:49.601]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2ee70ff5faa429c5 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2ee70ff5faa429c5"}
10:00:49 [10:00:49.602]
INFO:
prover-client:proving-broker Proving Broker started
10:00:49 [10:00:49.602]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e0787b5bc563e52a err=test error
{"provingJoId":"1:BASE_PARITY:e0787b5bc563e52a"}
10:00:49 [10:00:49.603]
INFO:
prover-client:proving-broker Proving Broker started
10:00:49 [10:00:49.604]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2ecfa0f905b59604 not found
{"provingJobId":"1:BASE_PARITY:2ecfa0f905b59604"}
10:00:49 [10:00:49.605]
INFO:
prover-client:proving-broker Proving Broker started
10:00:49 [10:00:49.605]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a734f72aa31052e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a734f72aa31052e3"}
10:00:49 [10:00:49.606]
INFO:
prover-client:proving-broker Proving Broker started
10:00:49 [10:00:49.607]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2661e86374694b56 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2661e86374694b56"}
10:00:49 [10:00:49.709]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2661e86374694b56 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2661e86374694b56"}
10:00:49 [10:00:49.716]
INFO:
prover-client:proving-broker Proving Broker started
10:00:49 [10:00:49.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0f003308f91f206 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0f003308f91f206"}
10:00:49 [10:00:49.819]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b0f003308f91f206 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b0f003308f91f206"}
10:00:49 [10:00:49.826]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d97e60e50cbfd566 epochNumber=10
{"provingJobId":"1:BASE_PARITY:d97e60e50cbfd566"}
10:00:49 [10:00:49.845]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:49 [10:00:49.956]
INFO:
prover-client:proving-broker Proving Broker started
10:00:49 [10:00:49.957]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e41012c8fad00b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e41012c8fad00b8"}
10:00:50 [10:00:50.108]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5e41012c8fad00b8 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5e41012c8fad00b8"}
10:00:50 [10:00:50.161]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.161]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b1cdea325087d18 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b1cdea325087d18"}
10:00:50 [10:00:50.162]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3b1cdea325087d18 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3b1cdea325087d18"}
10:00:50 [10:00:50.164]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.165]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7cadb186cf4fbf7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:7cadb186cf4fbf7d"}
10:00:50 [10:00:50.166]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:7cadb186cf4fbf7d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:7cadb186cf4fbf7d"}
10:00:50 [10:00:50.168]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7cadb186cf4fbf7d type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:7cadb186cf4fbf7d"}
10:00:50 [10:00:50.169]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.169]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc181d920c5f9239 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc181d920c5f9239"}
10:00:50 [10:00:50.170]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cc181d920c5f9239 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:cc181d920c5f9239"}
10:00:50 [10:00:50.171]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.172]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:825481ede46b3706 epochNumber=1
{"provingJobId":"1:BASE_PARITY:825481ede46b3706"}
10:00:50 [10:00:50.172]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:825481ede46b3706 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:825481ede46b3706"}
10:00:50 [10:00:50.173]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4750d0b24b83c93 epochNumber=3
{"provingJobId":"1:BASE_PARITY:c4750d0b24b83c93"}
10:00:50 [10:00:50.197]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:50 [10:00:50.199]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:825481ede46b3706 err=test error
{"provingJoId":"1:BASE_PARITY:825481ede46b3706"}
10:00:50 [10:00:50.201]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.201]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f3ac29d4edb6dbe8 settled=false
{"provingJobId":"1:BASE_PARITY:f3ac29d4edb6dbe8","status":"pending"}
10:00:50 [10:00:50.202]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ae795bae69c936ec settled=false
{"provingJobId":"1:BASE_PARITY:ae795bae69c936ec","status":"pending"}
10:00:50 [10:00:50.205]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.205]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:035a90324a760c7b settled=true
{"provingJobId":"1:BASE_PARITY:035a90324a760c7b","status":"fulfilled"}
10:00:50 [10:00:50.206]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:4a2d45f7793daac6 settled=true
{"provingJobId":"2:BASE_PARITY:4a2d45f7793daac6","status":"fulfilled"}
10:00:50 [10:00:50.207]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.207]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1eaa36a7b9cb32a7 settled=true
{"provingJobId":"1:BASE_PARITY:1eaa36a7b9cb32a7","status":"fulfilled"}
10:00:50 [10:00:50.208]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2f8fe5bb50cff705 settled=false
{"provingJobId":"1:BASE_PARITY:2f8fe5bb50cff705","status":"pending"}
10:00:50 [10:00:50.209]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.211]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f980d20a839da9b epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f980d20a839da9b"}
10:00:50 [10:00:50.213]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.213]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8782506c6a460632 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8782506c6a460632"}
10:00:50 [10:00:50.214]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:8782506c6a460632: Error: db error: Error: db error
10:00:50 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:8782506c6a460632"}
10:00:50 [10:00:50.216]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.217]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc08b43fc62f3b49 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc08b43fc62f3b49"}
10:00:50 [10:00:50.218]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dc08b43fc62f3b49 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dc08b43fc62f3b49"}
10:00:50 [10:00:50.218]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dc08b43fc62f3b49 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dc08b43fc62f3b49"}
10:00:50 [10:00:50.220]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.221]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8fde87442d7f073 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8fde87442d7f073"}
10:00:50 [10:00:50.222]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b8fde87442d7f073 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b8fde87442d7f073"}
10:00:50 [10:00:50.222]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b8fde87442d7f073 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b8fde87442d7f073"}
10:00:50 [10:00:50.223]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:b8fde87442d7f073: Error: db error
10:00:50 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:b8fde87442d7f073"}
10:00:50 [10:00:50.225]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.225]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03dd2ec4a408a050 epochNumber=1
{"provingJobId":"1:BASE_PARITY:03dd2ec4a408a050"}
10:00:50 [10:00:50.226]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:03dd2ec4a408a050 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:03dd2ec4a408a050"}
10:00:50 [10:00:50.226]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:03dd2ec4a408a050 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:03dd2ec4a408a050"}
10:00:50 [10:00:50.228]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.228]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21813f257b86809a epochNumber=1
{"provingJobId":"1:BASE_PARITY:21813f257b86809a"}
10:00:50 [10:00:50.228]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:21813f257b86809a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:21813f257b86809a"}
10:00:50 [10:00:50.229]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:21813f257b86809a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:21813f257b86809a"}
10:00:50 [10:00:50.229]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:21813f257b86809a jobErr=test error: Error: db error
10:00:50 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:21813f257b86809a"}
10:00:50 [10:00:50.231]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.231]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:54b8ba0630d365eb not found
{"provingJobId":"1:BASE_PARITY:54b8ba0630d365eb"}
10:00:50 [10:00:50.233]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.233]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:22417dec188092fa err=test error
{"provingJoId":"1:BASE_PARITY:22417dec188092fa"}
10:00:50 [10:00:50.235]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.245]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fdd0ba87416ef68d epochNumber=1
{"provingJobId":"1:BASE_PARITY:fdd0ba87416ef68d"}
10:00:50 [10:00:50.246]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fdd0ba87416ef68d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fdd0ba87416ef68d"}
10:00:50 [10:00:50.246]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fdd0ba87416ef68d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fdd0ba87416ef68d"}
10:00:50 [10:00:50.257]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:42cba0097cd926e8 epochNumber=2
{"provingJobId":"2:BASE_PARITY:42cba0097cd926e8"}
10:00:50 [10:00:50.258]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:42cba0097cd926e8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:42cba0097cd926e8"}
10:00:50 [10:00:50.258]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:42cba0097cd926e8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:42cba0097cd926e8"}
10:00:50 [10:00:50.319]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:e64586fdb6aa94ea epochNumber=3
{"provingJobId":"3:BASE_PARITY:e64586fdb6aa94ea"}
10:00:50 [10:00:50.336]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:50 [10:00:50.382]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:fb676fb4ce6e2297 epochNumber=4
{"provingJobId":"4:BASE_PARITY:fb676fb4ce6e2297"}
10:00:50 [10:00:50.388]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:50 [10:00:50.444]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:1cf9ea04e1ddc95d epochNumber=5
{"provingJobId":"5:BASE_PARITY:1cf9ea04e1ddc95d"}
10:00:50 [10:00:50.469]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:50 [10:00:50.500]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.501]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:208baf2c79b31a71 epochNumber=42
{"provingJobId":"1:BASE_PARITY:208baf2c79b31a71"}
10:00:50 [10:00:50.502]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testQ9pSLy/42 with map size 1073741824
10:00:50 [10:00:50.509]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.536]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:b3aa117c4f0bb645 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b3aa117c4f0bb645"}
10:00:50 [10:00:50.540]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bbe4f8093ab913c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bbe4f8093ab913c2"}
10:00:50 [10:00:50.543]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCVsr2Q/1 with map size 1073741824
10:00:50 [10:00:50.554]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.576]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5744c1c7e35f52d epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5744c1c7e35f52d"}
10:00:50 [10:00:50.587]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.588]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7954531031641f20 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7954531031641f20"}
10:00:50 [10:00:50.589]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testobJbsV/1 with map size 1073741824
10:00:50 [10:00:50.592]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.612]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7954531031641f20 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7954531031641f20"}
10:00:50 [10:00:50.617]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae4280da60bbad44 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae4280da60bbad44"}
10:00:50 [10:00:50.619]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDVQq2Q/1 with map size 1073741824
10:00:50 [10:00:50.621]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.645]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ae4280da60bbad44 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ae4280da60bbad44"}
10:00:50 [10:00:50.646]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ae4280da60bbad44 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ae4280da60bbad44"}
10:00:50 [10:00:50.657]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ae4280da60bbad44 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ae4280da60bbad44"}
10:00:50 [10:00:50.663]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.663]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:503d25bd656b9468 epochNumber=1
{"provingJobId":"1:BASE_PARITY:503d25bd656b9468"}
10:00:50 [10:00:50.664]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiHDWLa/1 with map size 1073741824
10:00:50 [10:00:50.667]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.690]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.693]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d13de1ba5f3d3f1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d13de1ba5f3d3f1"}
10:00:50 [10:00:50.694]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1HxdLK/1 with map size 1073741824
10:00:50 [10:00:50.697]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.716]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4d13de1ba5f3d3f1
{"provingJobId":"1:BASE_PARITY:4d13de1ba5f3d3f1"}
10:00:50 [10:00:50.717]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4d13de1ba5f3d3f1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4d13de1ba5f3d3f1"}
10:00:50 [10:00:50.717]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4d13de1ba5f3d3f1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4d13de1ba5f3d3f1"}
10:00:50 [10:00:50.727]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.728]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57988f38773b633d epochNumber=1
{"provingJobId":"1:BASE_PARITY:57988f38773b633d"}
10:00:50 [10:00:50.729]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVzTMAJ/1 with map size 1073741824
10:00:50 [10:00:50.731]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.755]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:57988f38773b633d
{"provingJobId":"1:BASE_PARITY:57988f38773b633d"}
10:00:50 [10:00:50.755]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:57988f38773b633d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:57988f38773b633d"}
10:00:50 [10:00:50.764]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.765]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5707b9db3a666c84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5707b9db3a666c84"}
10:00:50 [10:00:50.768]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNbpZoU/1 with map size 1073741824
10:00:50 [10:00:50.771]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.789]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5707b9db3a666c84 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5707b9db3a666c84"}
10:00:50 [10:00:50.789]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5707b9db3a666c84 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5707b9db3a666c84"}
10:00:50 [10:00:50.798]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.798]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bba6283558a8326d epochNumber=1
{"provingJobId":"1:BASE_PARITY:bba6283558a8326d"}
10:00:50 [10:00:50.800]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testonqoOs/1 with map size 1073741824
10:00:50 [10:00:50.802]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.825]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bba6283558a8326d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bba6283558a8326d"}
10:00:50 [10:00:50.825]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bba6283558a8326d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bba6283558a8326d"}
10:00:50 [10:00:50.836]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.837]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3d7fc14f6ff242f epochNumber=0
{"provingJobId":"1:BASE_PARITY:f3d7fc14f6ff242f"}
10:00:50 [10:00:50.837]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testJAP6N3/0 with map size 1073741824
10:00:50 [10:00:50.839]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.866]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f3d7fc14f6ff242f epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f3d7fc14f6ff242f"}
10:00:50 [10:00:50.866]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5c250b950032135 epochNumber=0
{"provingJobId":"1:BASE_PARITY:b5c250b950032135"}
10:00:50 [10:00:50.867]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f3d7fc14f6ff242f epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f3d7fc14f6ff242f"}
10:00:50 [10:00:50.867]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:234ca8abe50f3d63 epochNumber=0
{"provingJobId":"1:BASE_PARITY:234ca8abe50f3d63"}
10:00:50 [10:00:50.882]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.884]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.885]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:184d07ae47185f7b epochNumber=2
{"provingJobId":"1:BASE_PARITY:184d07ae47185f7b"}
10:00:50 [10:00:50.886]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testT1ZDQs/2 with map size 1073741824
10:00:50 [10:00:50.888]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.906]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f260eab9c4fe363a epochNumber=1
{"provingJobId":"1:BASE_PARITY:f260eab9c4fe363a"}
10:00:50 [10:00:50.907]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testT1ZDQs/1 with map size 1073741824
10:00:50 [10:00:50.909]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.929]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.930]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be17e39e6f1e986d epochNumber=1
{"provingJobId":"1:BASE_PARITY:be17e39e6f1e986d"}
10:00:50 [10:00:50.932]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6tfuDD/1 with map size 1073741824
10:00:50 [10:00:50.934]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.954]
INFO:
prover-client:proving-broker Proving Broker started
10:00:50 [10:00:50.955]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35104e9432b67752 epochNumber=1
{"provingJobId":"1:BASE_PARITY:35104e9432b67752"}
10:00:50 [10:00:50.956]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0sivUw/1 with map size 1073741824
10:00:50 [10:00:50.959]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:50 [10:00:50.976]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a736e189f01c27f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a736e189f01c27f0"}
10:00:50 [10:00:50.982]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d726201c9671a044 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d726201c9671a044"}
10:00:50 [10:00:50.983]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0sivUw/2 with map size 1073741824
10:00:50 [10:00:50.984]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.001]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3cac5c8077eeeca9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3cac5c8077eeeca9"}
10:00:51 [10:00:51.010]
INFO:
prover-client:proving-broker Proving Broker started
10:00:51 [10:00:51.011]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5809fd60d4dcf62d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5809fd60d4dcf62d"}
10:00:51 [10:00:51.012]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcI24ZG/1 with map size 1073741824
10:00:51 [10:00:51.014]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.032]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3d63fd7623f5c01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3d63fd7623f5c01"}
10:00:51 [10:00:51.038]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb19d8d906f46d8b epochNumber=2
{"provingJobId":"1:BASE_PARITY:eb19d8d906f46d8b"}
10:00:51 [10:00:51.039]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcI24ZG/2 with map size 1073741824
10:00:51 [10:00:51.041]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.058]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f4fcfbff0d939d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f4fcfbff0d939d9"}
10:00:51 [10:00:51.067]
INFO:
prover-client:proving-broker Proving Broker started
10:00:51 [10:00:51.068]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc08f5a6fbac2047 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc08f5a6fbac2047"}
10:00:51 [10:00:51.069]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6Yl9pu/1 with map size 1073741824
10:00:51 [10:00:51.070]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.085]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06fae55c365b856a epochNumber=1
{"provingJobId":"1:BASE_PARITY:06fae55c365b856a"}
10:00:51 [10:00:51.091]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ca0691554ff3df8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7ca0691554ff3df8"}
10:00:51 [10:00:51.092]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6Yl9pu/2 with map size 1073741824
10:00:51 [10:00:51.094]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.110]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:207806bd53399f5b epochNumber=1
{"provingJobId":"1:BASE_PARITY:207806bd53399f5b"}
10:00:51 [10:00:51.118]
INFO:
prover-client:proving-broker Proving Broker started
10:00:51 [10:00:51.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:906bd5fd77c4333f epochNumber=1
{"provingJobId":"1:BASE_PARITY:906bd5fd77c4333f"}
10:00:51 [10:00:51.120]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt8dejr/1 with map size 1073741824
10:00:51 [10:00:51.121]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.138]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea9c33a02996f471 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea9c33a02996f471"}
10:00:51 [10:00:51.144]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:906bd5fd77c4333f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:906bd5fd77c4333f"}
10:00:51 [10:00:51.154]
INFO:
prover-client:proving-broker Proving Broker started
10:00:51 [10:00:51.154]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d10fe08fe6eeed5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:d10fe08fe6eeed5d"}
10:00:51 [10:00:51.155]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test66tM1Z/1 with map size 1073741824
10:00:51 [10:00:51.156]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.172]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55fab039fbcb07d8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:55fab039fbcb07d8"}
10:00:51 [10:00:51.178]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d10fe08fe6eeed5d type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:d10fe08fe6eeed5d"}
10:00:51 [10:00:51.186]
INFO:
prover-client:proving-broker Proving Broker started
10:00:51 [10:00:51.187]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5cdbd39fa8ee6f66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5cdbd39fa8ee6f66"}
10:00:51 [10:00:51.187]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzROg4V/1 with map size 1073741824
10:00:51 [10:00:51.189]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.207]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:213e2d3fffdb7f7e epochNumber=1
{"provingJobId":"1:BASE_PARITY:213e2d3fffdb7f7e"}
10:00:51 [10:00:51.214]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5cdbd39fa8ee6f66 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:5cdbd39fa8ee6f66"}
10:00:51 [10:00:51.217]
INFO:
prover-client:proving-broker Proving Broker started
10:00:51 [10:00:51.218]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b05f4635b7ac34b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b05f4635b7ac34b"}
10:00:51 [10:00:51.219]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzdaFF6/1 with map size 1073741824
10:00:51 [10:00:51.221]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.238]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4b05f4635b7ac34b
{"provingJobId":"1:BASE_PARITY:4b05f4635b7ac34b"}
10:00:51 [10:00:51.238]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4b05f4635b7ac34b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4b05f4635b7ac34b"}
10:00:51 [10:00:51.244]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88486e071d867b99 epochNumber=1
{"provingJobId":"1:BASE_PARITY:88486e071d867b99"}
10:00:51 [10:00:51.250]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4b05f4635b7ac34b has already been completed
{"provingJobId":"1:BASE_PARITY:4b05f4635b7ac34b"}
10:00:51 [10:00:51.253]
INFO:
prover-client:proving-broker Proving Broker started
10:00:51 [10:00:51.254]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ac5a70dc76620511 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ac5a70dc76620511"}
10:00:51 [10:00:51.255]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testW23Sco/1 with map size 1073741824
10:00:51 [10:00:51.256]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:51 [10:00:51.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0052d99bd03a67d6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0052d99bd03a67d6"}
10:00:51 [10:00:51.273]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testW23Sco/2 with map size 1073741824
10:00:51 [10:00:51.275]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:52 [10:00:52.344]
INFO:
prover-client:proving-broker Proving Broker started
10:00:52 [10:00:52.353]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ac5a70dc76620511 settled=false
{"provingJobId":"1:BASE_PARITY:ac5a70dc76620511","status":"pending"}
10:00:52 [10:00:52.355]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0052d99bd03a67d6 settled=false
{"provingJobId":"1:BASE_PARITY:0052d99bd03a67d6","status":"pending"}
10:00:52 [10:00:52.356]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ac5a70dc76620511 type=BASE_PARITY startedAt=1750240851293 older agent has taken job
{"provingJobId":"1:BASE_PARITY:ac5a70dc76620511"}
10:00:52 [10:00:52.357]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ac5a70dc76620511 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:ac5a70dc76620511"}
10:00:52 [10:00:52.362]
INFO:
prover-client:proving-broker Proving Broker started
10:00:52 [10:00:52.363]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82d6b32e76354c54 epochNumber=1
{"provingJobId":"1:BASE_PARITY:82d6b32e76354c54"}
10:00:52 [10:00:52.363]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJqXPpY/1 with map size 1073741824
10:00:52 [10:00:52.365]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:52 [10:00:52.384]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50bc93502feae03f epochNumber=2
{"provingJobId":"1:BASE_PARITY:50bc93502feae03f"}
10:00:52 [10:00:52.385]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJqXPpY/2 with map size 1073741824
10:00:52 [10:00:52.387]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:53 [10:00:53.404]
INFO:
prover-client:proving-broker Proving Broker started
10:00:53 [10:00:53.406]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:82d6b32e76354c54 settled=false
{"provingJobId":"1:BASE_PARITY:82d6b32e76354c54","status":"pending"}
10:00:53 [10:00:53.407]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:50bc93502feae03f settled=false
{"provingJobId":"1:BASE_PARITY:50bc93502feae03f","status":"pending"}
10:00:53 [10:00:53.408]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:82d6b32e76354c54 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:82d6b32e76354c54"}
10:00:53 [10:00:53.413]
INFO:
prover-client:proving-broker Proving Broker started
10:00:53 [10:00:53.413]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b3f2d7bf27dbca34 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b3f2d7bf27dbca34"}
10:00:53 [10:00:53.414]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuUha2z/1 with map size 1073741824
10:00:53 [10:00:53.419]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:53 [10:00:53.438]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c74562fa5a5b2476 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c74562fa5a5b2476"}
10:00:53 [10:00:53.439]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testuUha2z/2 with map size 1073741824
10:00:53 [10:00:53.446]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:54 [10:00:54.466]
INFO:
prover-client:proving-broker Proving Broker started
10:00:54 [10:00:54.468]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b3f2d7bf27dbca34 settled=false
{"provingJobId":"1:BASE_PARITY:b3f2d7bf27dbca34","status":"pending"}
10:00:54 [10:00:54.469]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c74562fa5a5b2476 settled=false
{"provingJobId":"1:BASE_PARITY:c74562fa5a5b2476","status":"pending"}
10:00:54 [10:00:54.471]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b3f2d7bf27dbca34 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b3f2d7bf27dbca34"}
10:00:54 [10:00:54.471]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b3f2d7bf27dbca34 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b3f2d7bf27dbca34"}
10:00:54 [10:00:54.483]
INFO:
prover-client:proving-broker Proving Broker started
10:00:54 [10:00:54.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3428033afdf9f0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3428033afdf9f0b"}
10:00:54 [10:00:54.485]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVq4wPb/1 with map size 1073741824
10:00:54 [10:00:54.488]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:54 [10:00:54.504]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:535fbee2288cde87 epochNumber=2
{"provingJobId":"1:BASE_PARITY:535fbee2288cde87"}
10:00:54 [10:00:54.505]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVq4wPb/2 with map size 1073741824
10:00:54 [10:00:54.507]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:54 [10:00:54.523]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a3428033afdf9f0b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a3428033afdf9f0b"}
10:00:54 [10:00:54.529]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:535fbee2288cde87 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:535fbee2288cde87"}
10:00:54 [10:00:54.538]
INFO:
prover-client:proving-broker Proving Broker started
10:00:54 [10:00:54.538]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96c0a59ad99e03da epochNumber=1
{"provingJobId":"1:BASE_PARITY:96c0a59ad99e03da"}
10:00:54 [10:00:54.539]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDODJak/1 with map size 1073741824
10:00:54 [10:00:54.548]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:54 [10:00:54.567]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef36f4a40dca143a epochNumber=2
{"provingJobId":"1:BASE_PARITY:ef36f4a40dca143a"}
10:00:54 [10:00:54.568]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDODJak/2 with map size 1073741824
10:00:54 [10:00:54.570]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:54 [10:00:54.586]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:96c0a59ad99e03da type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:96c0a59ad99e03da"}
10:00:54 [10:00:54.587]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:96c0a59ad99e03da type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:96c0a59ad99e03da"}
10:00:54 [10:00:54.593]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ef36f4a40dca143a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ef36f4a40dca143a"}
10:00:54 [10:00:54.604]
INFO:
prover-client:proving-broker Proving Broker started
10:00:54 [10:00:54.605]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6a32b53a5b27fc8e err=test error
{"provingJoId":"1:BASE_PARITY:6a32b53a5b27fc8e"}
10:00:54 [10:00:54.609]
INFO:
prover-client:proving-broker Proving Broker started
10:00:54 [10:00:54.610]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5f2726eb5d5d4ace not found
{"provingJobId":"1:BASE_PARITY:5f2726eb5d5d4ace"}
10:00:54 [10:00:54.613]
INFO:
prover-client:proving-broker Proving Broker started
10:00:54 [10:00:54.614]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5473e68565c269d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5473e68565c269d9"}
10:00:54 [10:00:54.615]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvh7RdN/1 with map size 1073741824
10:00:54 [10:00:54.617]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:54 [10:00:54.644]
INFO:
prover-client:proving-broker Proving Broker started
10:00:54 [10:00:54.645]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:394087f54322f1be epochNumber=1
{"provingJobId":"1:BASE_PARITY:394087f54322f1be"}
10:00:54 [10:00:54.646]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqEdl8i/1 with map size 1073741824
10:00:54 [10:00:54.651]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:54 [10:00:54.772]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:394087f54322f1be timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:394087f54322f1be"}
10:00:54 [10:00:54.776]
INFO:
prover-client:proving-broker Proving Broker started
10:00:54 [10:00:54.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc93ad6f44485449 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc93ad6f44485449"}
10:00:54 [10:00:54.779]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9Kry0Y/1 with map size 1073741824
10:00:54 [10:00:54.781]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:54 [10:00:54.906]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dc93ad6f44485449 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:dc93ad6f44485449"}
10:00:54 [10:00:54.910]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2763749bf9581df4 epochNumber=10
{"provingJobId":"1:BASE_PARITY:2763749bf9581df4"}
10:00:54 [10:00:54.911]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test9Kry0Y/10 with map size 1073741824
10:00:54 [10:00:54.913]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:54 [10:00:54.932]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:54 [10:00:54.933]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:00:55 [10:00:55.062]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9afeb5e9000dcab0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9afeb5e9000dcab0"}
10:00:55 [10:00:55.064]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHfoVfJ/1 with map size 1073741824
10:00:55 [10:00:55.066]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.238]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9afeb5e9000dcab0 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9afeb5e9000dcab0"}
10:00:55 [10:00:55.291]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.292]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48b9e060eb32e066 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48b9e060eb32e066"}
10:00:55 [10:00:55.294]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaGXmBC/1 with map size 1073741824
10:00:55 [10:00:55.300]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.322]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:48b9e060eb32e066 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:48b9e060eb32e066"}
10:00:55 [10:00:55.328]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.329]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dadb07d9b1a467cc epochNumber=1
{"provingJobId":"1:BASE_PARITY:dadb07d9b1a467cc"}
10:00:55 [10:00:55.331]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKpqaU2/1 with map size 1073741824
10:00:55 [10:00:55.333]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.354]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:dadb07d9b1a467cc type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:dadb07d9b1a467cc"}
10:00:55 [10:00:55.355]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dadb07d9b1a467cc type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:dadb07d9b1a467cc"}
10:00:55 [10:00:55.366]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.368]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05db7fd88dca9d78 epochNumber=1
{"provingJobId":"1:BASE_PARITY:05db7fd88dca9d78"}
10:00:55 [10:00:55.370]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testizwsAm/1 with map size 1073741824
10:00:55 [10:00:55.373]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.394]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:05db7fd88dca9d78 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:05db7fd88dca9d78"}
10:00:55 [10:00:55.404]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.405]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f16b395e28320e5f epochNumber=1
{"provingJobId":"1:BASE_PARITY:f16b395e28320e5f"}
10:00:55 [10:00:55.406]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testc089Zk/1 with map size 1073741824
10:00:55 [10:00:55.408]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.425]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f16b395e28320e5f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f16b395e28320e5f"}
10:00:55 [10:00:55.426]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54fe341314f6b394 epochNumber=3
{"provingJobId":"1:BASE_PARITY:54fe341314f6b394"}
10:00:55 [10:00:55.427]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testc089Zk/3 with map size 1073741824
10:00:55 [10:00:55.429]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.454]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:55 [10:00:55.455]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:00:55 [10:00:55.474]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f16b395e28320e5f err=test error
{"provingJoId":"1:BASE_PARITY:f16b395e28320e5f"}
10:00:55 [10:00:55.481]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6xK7HC/1 with map size 1073741824
10:00:55 [10:00:55.484]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.504]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6xK7HC/2 with map size 1073741824
10:00:55 [10:00:55.507]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.528]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.530]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0b0d6d6bec6dbd20 settled=false
{"provingJobId":"1:BASE_PARITY:0b0d6d6bec6dbd20","status":"pending"}
10:00:55 [10:00:55.531]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4ed21ba1c5102091 settled=false
{"provingJobId":"1:BASE_PARITY:4ed21ba1c5102091","status":"pending"}
10:00:55 [10:00:55.537]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQU7Vmm/1 with map size 1073741824
10:00:55 [10:00:55.539]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.565]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQU7Vmm/2 with map size 1073741824
10:00:55 [10:00:55.569]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.598]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.601]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1b4a628984679fd4 settled=true
{"provingJobId":"1:BASE_PARITY:1b4a628984679fd4","status":"fulfilled"}
10:00:55 [10:00:55.602]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:61cd2b3a1123b5f7 settled=true
{"provingJobId":"2:BASE_PARITY:61cd2b3a1123b5f7","status":"fulfilled"}
10:00:55 [10:00:55.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMi7EGp/1 with map size 1073741824
10:00:55 [10:00:55.611]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.637]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMi7EGp/2 with map size 1073741824
10:00:55 [10:00:55.639]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.658]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.660]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dc6dbc82560c48ca settled=true
{"provingJobId":"1:BASE_PARITY:dc6dbc82560c48ca","status":"fulfilled"}
10:00:55 [10:00:55.662]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d84c48995bb72d23 settled=false
{"provingJobId":"1:BASE_PARITY:d84c48995bb72d23","status":"pending"}
10:00:55 [10:00:55.668]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.669]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ad9e1db957079a0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:ad9e1db957079a0f"}
10:00:55 [10:00:55.670]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzuNfht/1 with map size 1073741824
10:00:55 [10:00:55.673]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.697]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.698]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90002b73e917037b epochNumber=1
{"provingJobId":"1:BASE_PARITY:90002b73e917037b"}
10:00:55 [10:00:55.698]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:90002b73e917037b: Error: db error: Error: db error
10:00:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:90002b73e917037b"}
10:00:55 [10:00:55.703]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.703]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0e59b6ba6ad9949 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f0e59b6ba6ad9949"}
10:00:55 [10:00:55.704]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testd2WMTR/1 with map size 1073741824
10:00:55 [10:00:55.706]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.722]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f0e59b6ba6ad9949 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f0e59b6ba6ad9949"}
10:00:55 [10:00:55.722]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f0e59b6ba6ad9949 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f0e59b6ba6ad9949"}
10:00:55 [10:00:55.733]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.734]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9cf0112b6e6f274d epochNumber=1
{"provingJobId":"1:BASE_PARITY:9cf0112b6e6f274d"}
10:00:55 [10:00:55.736]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHDuziU/1 with map size 1073741824
10:00:55 [10:00:55.740]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.763]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9cf0112b6e6f274d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9cf0112b6e6f274d"}
10:00:55 [10:00:55.763]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9cf0112b6e6f274d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9cf0112b6e6f274d"}
10:00:55 [10:00:55.764]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:9cf0112b6e6f274d: Error: db error
10:00:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:9cf0112b6e6f274d"}
10:00:55 [10:00:55.768]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91720f224d86a404 epochNumber=1
{"provingJobId":"1:BASE_PARITY:91720f224d86a404"}
10:00:55 [10:00:55.770]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testd8RncE/1 with map size 1073741824
10:00:55 [10:00:55.773]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.796]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:91720f224d86a404 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:91720f224d86a404"}
10:00:55 [10:00:55.797]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:91720f224d86a404 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:91720f224d86a404"}
10:00:55 [10:00:55.809]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.810]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ba322b89cd0938c epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ba322b89cd0938c"}
10:00:55 [10:00:55.810]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbgy24j/1 with map size 1073741824
10:00:55 [10:00:55.814]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.838]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2ba322b89cd0938c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2ba322b89cd0938c"}
10:00:55 [10:00:55.839]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2ba322b89cd0938c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2ba322b89cd0938c"}
10:00:55 [10:00:55.840]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:2ba322b89cd0938c jobErr=test error: Error: db error
10:00:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:2ba322b89cd0938c"}
10:00:55 [10:00:55.845]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.845]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fad6a9ff6a5dbb28 not found
{"provingJobId":"1:BASE_PARITY:fad6a9ff6a5dbb28"}
10:00:55 [10:00:55.849]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.850]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:9bd0780c6ada821c err=test error
{"provingJoId":"1:BASE_PARITY:9bd0780c6ada821c"}
10:00:55 [10:00:55.853]
INFO:
prover-client:proving-broker Proving Broker started
10:00:55 [10:00:55.865]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3bf14c97ab2518e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3bf14c97ab2518e9"}
10:00:55 [10:00:55.866]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testumAKWr/1 with map size 1073741824
10:00:55 [10:00:55.867]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.886]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3bf14c97ab2518e9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3bf14c97ab2518e9"}
10:00:55 [10:00:55.886]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3bf14c97ab2518e9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3bf14c97ab2518e9"}
10:00:55 [10:00:55.905]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:018c96f86784ba18 epochNumber=2
{"provingJobId":"2:BASE_PARITY:018c96f86784ba18"}
10:00:55 [10:00:55.906]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testumAKWr/2 with map size 1073741824
10:00:55 [10:00:55.910]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:55 [10:00:55.932]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:018c96f86784ba18 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:018c96f86784ba18"}
10:00:55 [10:00:55.933]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:018c96f86784ba18 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:018c96f86784ba18"}
10:00:56 [10:00:56.005]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:984d18917e2be3d1 epochNumber=3
{"provingJobId":"3:BASE_PARITY:984d18917e2be3d1"}
10:00:56 [10:00:56.007]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testumAKWr/3 with map size 1073741824
10:00:56 [10:00:56.012]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:56 [10:00:56.036]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:56 [10:00:56.036]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:00:56 [10:00:56.091]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:d188051f9121debf epochNumber=4
{"provingJobId":"4:BASE_PARITY:d188051f9121debf"}
10:00:56 [10:00:56.092]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testumAKWr/4 with map size 1073741824
10:00:56 [10:00:56.094]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:56 [10:00:56.139]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:56 [10:00:56.139]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
10:00:56 [10:00:56.177]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:c697aea6a07f0b83 epochNumber=5
{"provingJobId":"5:BASE_PARITY:c697aea6a07f0b83"}
10:00:56 [10:00:56.178]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testumAKWr/5 with map size 1073741824
10:00:56 [10:00:56.180]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:00:56 [10:00:56.224]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:00:56 [10:00:56.224]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
10:00:56
PASS src/proving_broker/proving_broker.test.ts (
13.065 s)
10:00:56 ProvingBroker
10:00:56 Producer API
10:00:56
✓ refuses stale jobs (41 ms)
10:00:56
✓ enqueues jobs (3 ms)
10:00:56
✓ ignores duplicate jobs (3 ms)
10:00:56
✓ reports correct status when enqueuing repeat jobs (5 ms)
10:00:56
✓ throws an error in case of duplicate job IDs (27 ms)
10:00:56
✓ returns not-found status for non-existing jobs (2 ms)
10:00:56
✓ cancels jobs in queue (4 ms)
10:00:56
✓ cancels jobs in-progress (2 ms)
10:00:56
✓ returns job result if successful (2 ms)
10:00:56
✓ returns job error if failed (2 ms)
10:00:56
✓ correctly returns job status for concurrent writes (3 ms)
10:00:56
✓ refuses stale jobs (40 ms)
10:00:56
✓ enqueues jobs (46 ms)
10:00:56
✓ ignores duplicate jobs (30 ms)
10:00:56
✓ reports correct status when enqueuing repeat jobs (45 ms)
10:00:56
✓ throws an error in case of duplicate job IDs (27 ms)
10:00:56
✓ returns not-found status for non-existing jobs (3 ms)
10:00:56
✓ cancels jobs in queue (34 ms)
10:00:56
✓ cancels jobs in-progress (36 ms)
10:00:56
✓ returns job result if successful (34 ms)
10:00:56
✓ returns job error if failed (38 ms)
10:00:56
✓ correctly returns job status for concurrent writes (46 ms)
10:00:56 Consumer API
10:00:56
✓ returns undefined if no jobs are available (9 ms)
10:00:56
✓ returns jobs in priority order (2 ms)
10:00:56
✓ returns undefined if no jobs are available for the given allowList (2 ms)
10:00:56
✓ returns a job if it is in the allowList (2 ms)
10:00:56
✓ returns the most important job if it is in the allowList (3 ms)
10:00:56
✓ returns any job if filter is empty (2 ms)
10:00:56
✓ returns a new job when reporting job success (3 ms)
10:00:56
✓ returns a new job when reporting permanent error (2 ms)
10:00:56
✓ returns a new job when reporting retry-able error (2 ms)
10:00:56
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
10:00:56
✓ returns a new job if job is already in progress elsewhere (1060 ms)
10:00:56
✓ avoids sending the same job to a new agent after a restart (1005 ms)
10:00:56
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
10:00:56
✓ tracks job result if in progress (2 ms)
10:00:56
✓ tracks job result even if job is in queue (2 ms)
10:00:56
✓ ignores reported job error if unknown job (1 ms)
10:00:56
✓ ignores job result if unknown job (1 ms)
10:00:56
✓ returns undefined if no jobs are available (2 ms)
10:00:56
✓ returns jobs in priority order (45 ms)
10:00:56
✓ returns undefined if no jobs are available for the given allowList (24 ms)
10:00:56
✓ returns a job if it is in the allowList (56 ms)
10:00:56
✓ returns the most important job if it is in the allowList (56 ms)
10:00:56
✓ returns any job if filter is empty (51 ms)
10:00:56
✓ returns a new job when reporting job success (33 ms)
10:00:56
✓ returns a new job when reporting permanent error (33 ms)
10:00:56
✓ returns a new job when reporting retry-able error (30 ms)
10:00:56
✓ returns a new job when reporting progress if current one is cancelled (35 ms)
10:00:56
✓ returns a new job if job is already in progress elsewhere (1107 ms)
10:00:56
✓ avoids sending the same job to a new agent after a restart (1052 ms)
10:00:56
✓ avoids sending a completed job to a new agent after a restart (1068 ms)
10:00:56
✓ tracks job result if in progress (57 ms)
10:00:56
✓ tracks job result even if job is in queue (65 ms)
10:00:56
✓ ignores reported job error if unknown job (5 ms)
10:00:56
✓ ignores job result if unknown job (4 ms)
10:00:56 Timeouts
10:00:56
✓ tracks in progress jobs (1 ms)
10:00:56
✓ re-enqueues jobs that time out (109 ms)
10:00:56
✓ cancel stale jobs that time out (239 ms)
10:00:56
✓ keeps the jobs in progress while it is alive (205 ms)
10:00:56
✓ tracks in progress jobs (26 ms)
10:00:56
✓ re-enqueues jobs that time out (134 ms)
10:00:56
✓ cancel stale jobs that time out (285 ms)
10:00:56
✓ keeps the jobs in progress while it is alive (229 ms)
10:00:56 Retries
10:00:56
✓ retries jobs (3 ms)
10:00:56
✓ retries up to a maximum number of times (4 ms)
10:00:56
✓ passing retry=false does not retry (2 ms)
10:00:56
✓ does not retry if job is stale (29 ms)
10:00:56
✓ retries jobs (37 ms)
10:00:56
✓ retries up to a maximum number of times (35 ms)
10:00:56
✓ passing retry=false does not retry (39 ms)
10:00:56
✓ does not retry if job is stale (73 ms)
10:00:56 Database management
10:00:56
✓ re-enqueues proof requests on start (4 ms)
10:00:56
✓ restores proof results on start (2 ms)
10:00:56
✓ only re-enqueues unfinished jobs (2 ms)
10:00:56
✓ saves job when enqueued (3 ms)
10:00:56
✓ does not retain job if database fails to save (3 ms)
10:00:56
✓ saves job result (4 ms)
10:00:56
✓ saves result even if database fails to save (4 ms)
10:00:56
✓ saves job error (3 ms)
10:00:56
✓ saves job error even if database fails to save (3 ms)
10:00:56
✓ does not save job result if job is unknown (2 ms)
10:00:56
✓ does not save job error if job is unknown (2 ms)
10:00:56
✓ cleans up old jobs periodically (263 ms)
10:00:56
✓ re-enqueues proof requests on start (58 ms)
10:00:56
✓ restores proof results on start (69 ms)
10:00:56
✓ only re-enqueues unfinished jobs (60 ms)
10:00:56
✓ saves job when enqueued (29 ms)
10:00:56
✓ does not retain job if database fails to save (5 ms)
10:00:56
✓ saves job result (29 ms)
10:00:56
✓ saves result even if database fails to save (36 ms)
10:00:56
✓ saves job error (40 ms)
10:00:56
✓ saves job error even if database fails to save (35 ms)
10:00:56
✓ does not save job result if job is unknown (5 ms)
10:00:56
✓ does not save job error if job is unknown (4 ms)
10:00:56
✓ cleans up old jobs periodically (407 ms)
10:00:56
10:00:56
Test Suites: 1 passed, 1 total
10:00:56
Tests: 96 passed, 96 total
10:00:56
Snapshots: 0 total
10:00:56
Time: 13.166 s
10:00:56
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
10:00:56
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?