Parent Log:
http://ci.aztec-labs.com/3798e80704a63f3d
Command: 6db87052395d3483 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/41cc1f518fed92110a82ce3c59385a3f5c0d8817
Env: REF_NAME=gh-readonly-queue/next/pr-15041-9cf4a24266245fa3ca003adb00e8ae759880c107 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 16 17:16:31 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15041_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
17:16:36 [17:16:36.480]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.488]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eaeabaeaa7f945b6 epochNumber=42
{"provingJobId":"1:BASE_PARITY:eaeabaeaa7f945b6"}
17:16:36 [17:16:36.491]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:be21d528c5b36d26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:be21d528c5b36d26"}
17:16:36 [17:16:36.534]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.536]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a8bd7128bb01e014 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a8bd7128bb01e014"}
17:16:36 [17:16:36.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e52df7c37aea5b3d epochNumber=1
{"provingJobId":"1:BASE_PARITY:e52df7c37aea5b3d"}
17:16:36 [17:16:36.540]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:171b1ca6be45f975 epochNumber=1
{"provingJobId":"1:BASE_PARITY:171b1ca6be45f975"}
17:16:36 [17:16:36.542]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:171b1ca6be45f975 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:171b1ca6be45f975"}
17:16:36 [17:16:36.544]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.545]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90a6d754e879fb38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:90a6d754e879fb38"}
17:16:36 [17:16:36.547]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:90a6d754e879fb38 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:90a6d754e879fb38"}
17:16:36 [17:16:36.548]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:90a6d754e879fb38 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:90a6d754e879fb38"}
17:16:36 [17:16:36.548]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:90a6d754e879fb38 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:90a6d754e879fb38"}
17:16:36 [17:16:36.550]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.550]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3dc47e3ac01b85e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3dc47e3ac01b85e0"}
17:16:36 [17:16:36.582]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.585]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.585]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d5f05d9de80bb7e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d5f05d9de80bb7e5"}
17:16:36 [17:16:36.586]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d5f05d9de80bb7e5
{"provingJobId":"1:BASE_PARITY:d5f05d9de80bb7e5"}
17:16:36 [17:16:36.586]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d5f05d9de80bb7e5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d5f05d9de80bb7e5"}
17:16:36 [17:16:36.587]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d5f05d9de80bb7e5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:d5f05d9de80bb7e5"}
17:16:36 [17:16:36.589]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.589]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1993a6ba99332e6e epochNumber=1
{"provingJobId":"1:BASE_PARITY:1993a6ba99332e6e"}
17:16:36 [17:16:36.590]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1993a6ba99332e6e
{"provingJobId":"1:BASE_PARITY:1993a6ba99332e6e"}
17:16:36 [17:16:36.590]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1993a6ba99332e6e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1993a6ba99332e6e"}
17:16:36 [17:16:36.592]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.593]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:108a449e8f0e9153 epochNumber=1
{"provingJobId":"1:BASE_PARITY:108a449e8f0e9153"}
17:16:36 [17:16:36.593]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:108a449e8f0e9153 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:108a449e8f0e9153"}
17:16:36 [17:16:36.594]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:108a449e8f0e9153 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:108a449e8f0e9153"}
17:16:36 [17:16:36.595]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.596]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de1760b807db5bbc epochNumber=1
{"provingJobId":"1:BASE_PARITY:de1760b807db5bbc"}
17:16:36 [17:16:36.596]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:de1760b807db5bbc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:de1760b807db5bbc"}
17:16:36 [17:16:36.596]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:de1760b807db5bbc type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:de1760b807db5bbc"}
17:16:36 [17:16:36.597]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.598]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67c6acd18f949729 epochNumber=0
{"provingJobId":"1:BASE_PARITY:67c6acd18f949729"}
17:16:36 [17:16:36.598]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:67c6acd18f949729 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:67c6acd18f949729"}
17:16:36 [17:16:36.599]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5e0ed8bafa0b727 epochNumber=0
{"provingJobId":"1:BASE_PARITY:f5e0ed8bafa0b727"}
17:16:36 [17:16:36.599]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:67c6acd18f949729 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:67c6acd18f949729"}
17:16:36 [17:16:36.599]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b58683bb4384f44 epochNumber=0
{"provingJobId":"1:BASE_PARITY:4b58683bb4384f44"}
17:16:36 [17:16:36.601]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.602]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.602]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:30b5b3880f4f3fb6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:30b5b3880f4f3fb6"}
17:16:36 [17:16:36.602]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50a5f93d3f289dc4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:50a5f93d3f289dc4"}
17:16:36 [17:16:36.604]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.604]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f05ec3e6118bece epochNumber=1
{"provingJobId":"1:BASE_PARITY:7f05ec3e6118bece"}
17:16:36 [17:16:36.605]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.605]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bedef187cb52570f epochNumber=1
{"provingJobId":"1:BASE_PARITY:bedef187cb52570f"}
17:16:36 [17:16:36.605]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46e757ab57e2faba epochNumber=1
{"provingJobId":"1:BASE_PARITY:46e757ab57e2faba"}
17:16:36 [17:16:36.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d00aced4b7c23b80 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d00aced4b7c23b80"}
17:16:36 [17:16:36.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7bde1e3dc4677558 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7bde1e3dc4677558"}
17:16:36 [17:16:36.607]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.607]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb1fda7596ca40da epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb1fda7596ca40da"}
17:16:36 [17:16:36.607]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09ca3efa9d313ad9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:09ca3efa9d313ad9"}
17:16:36 [17:16:36.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c80da220ec860a3e epochNumber=2
{"provingJobId":"1:BASE_PARITY:c80da220ec860a3e"}
17:16:36 [17:16:36.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee6a3997d2dcadec epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee6a3997d2dcadec"}
17:16:36 [17:16:36.609]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.609]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f039d10263d3cfa epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f039d10263d3cfa"}
17:16:36 [17:16:36.609]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:582b59902fe1580f epochNumber=1
{"provingJobId":"1:BASE_PARITY:582b59902fe1580f"}
17:16:36 [17:16:36.609]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf3cbe1db052ff8c epochNumber=2
{"provingJobId":"1:BASE_PARITY:bf3cbe1db052ff8c"}
17:16:36 [17:16:36.610]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c43aa2d0953d71b epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c43aa2d0953d71b"}
17:16:36 [17:16:36.610]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.611]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a74fba115b1c70e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a74fba115b1c70e"}
17:16:36 [17:16:36.611]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aab86f06e817e024 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aab86f06e817e024"}
17:16:36 [17:16:36.611]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4a74fba115b1c70e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4a74fba115b1c70e"}
17:16:36 [17:16:36.613]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.613]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b3ffc76d705bda6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b3ffc76d705bda6"}
17:16:36 [17:16:36.614]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a939448bddfbc7a epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a939448bddfbc7a"}
17:16:36 [17:16:36.614]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4b3ffc76d705bda6 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:4b3ffc76d705bda6"}
17:16:36 [17:16:36.615]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.615]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6dd8c9be5a2e8c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6dd8c9be5a2e8c0"}
17:16:36 [17:16:36.616]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42d20b86dc76cf62 epochNumber=1
{"provingJobId":"1:BASE_PARITY:42d20b86dc76cf62"}
17:16:36 [17:16:36.616]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c6dd8c9be5a2e8c0 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:c6dd8c9be5a2e8c0"}
17:16:36 [17:16:36.617]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.617]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:37936372636d0392 epochNumber=1
{"provingJobId":"1:BASE_PARITY:37936372636d0392"}
17:16:36 [17:16:36.618]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:37936372636d0392
{"provingJobId":"1:BASE_PARITY:37936372636d0392"}
17:16:36 [17:16:36.618]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:37936372636d0392 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:37936372636d0392"}
17:16:36 [17:16:36.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e1154f9b267745a epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e1154f9b267745a"}
17:16:36 [17:16:36.619]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:37936372636d0392 has already been completed
{"provingJobId":"1:BASE_PARITY:37936372636d0392"}
17:16:36 [17:16:36.625]
INFO:
prover-client:proving-broker Proving Broker started
17:16:36 [17:16:36.626]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb84ce820e6826a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb84ce820e6826a6"}
17:16:36 [17:16:36.627]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6702dce0bcdcf6a epochNumber=2
{"provingJobId":"1:BASE_PARITY:b6702dce0bcdcf6a"}
17:16:37 [17:16:37.679]
INFO:
prover-client:proving-broker Proving Broker started
17:16:37 [17:16:37.680]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:eb84ce820e6826a6 settled=false
{"provingJobId":"1:BASE_PARITY:eb84ce820e6826a6","status":"pending"}
17:16:37 [17:16:37.680]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b6702dce0bcdcf6a settled=false
{"provingJobId":"1:BASE_PARITY:b6702dce0bcdcf6a","status":"pending"}
17:16:37 [17:16:37.681]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb84ce820e6826a6 type=BASE_PARITY startedAt=1750094196627 older agent has taken job
{"provingJobId":"1:BASE_PARITY:eb84ce820e6826a6"}
17:16:37 [17:16:37.681]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb84ce820e6826a6 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:eb84ce820e6826a6"}
17:16:37 [17:16:37.682]
INFO:
prover-client:proving-broker Proving Broker started
17:16:37 [17:16:37.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f9213c2fd17374a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f9213c2fd17374a1"}
17:16:37 [17:16:37.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8cd70241f34842b epochNumber=2
{"provingJobId":"1:BASE_PARITY:b8cd70241f34842b"}
17:16:38 [17:16:38.684]
INFO:
prover-client:proving-broker Proving Broker started
17:16:38 [17:16:38.684]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f9213c2fd17374a1 settled=false
{"provingJobId":"1:BASE_PARITY:f9213c2fd17374a1","status":"pending"}
17:16:38 [17:16:38.685]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b8cd70241f34842b settled=false
{"provingJobId":"1:BASE_PARITY:b8cd70241f34842b","status":"pending"}
17:16:38 [17:16:38.685]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f9213c2fd17374a1 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:f9213c2fd17374a1"}
17:16:38 [17:16:38.687]
INFO:
prover-client:proving-broker Proving Broker started
17:16:38 [17:16:38.687]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9da6f24becadf46a epochNumber=1
{"provingJobId":"1:BASE_PARITY:9da6f24becadf46a"}
17:16:38 [17:16:38.688]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2accab2db833745d epochNumber=2
{"provingJobId":"1:BASE_PARITY:2accab2db833745d"}
17:16:39 [17:16:39.688]
INFO:
prover-client:proving-broker Proving Broker started
17:16:39 [17:16:39.689]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9da6f24becadf46a settled=false
{"provingJobId":"1:BASE_PARITY:9da6f24becadf46a","status":"pending"}
17:16:39 [17:16:39.689]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2accab2db833745d settled=false
{"provingJobId":"1:BASE_PARITY:2accab2db833745d","status":"pending"}
17:16:39 [17:16:39.690]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9da6f24becadf46a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9da6f24becadf46a"}
17:16:39 [17:16:39.690]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9da6f24becadf46a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9da6f24becadf46a"}
17:16:39 [17:16:39.692]
INFO:
prover-client:proving-broker Proving Broker started
17:16:39 [17:16:39.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:38f17f9b7a7b1ead epochNumber=1
{"provingJobId":"1:BASE_PARITY:38f17f9b7a7b1ead"}
17:16:39 [17:16:39.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d79a2a8b0b8fb21 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1d79a2a8b0b8fb21"}
17:16:39 [17:16:39.694]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:38f17f9b7a7b1ead type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:38f17f9b7a7b1ead"}
17:16:39 [17:16:39.694]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1d79a2a8b0b8fb21 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1d79a2a8b0b8fb21"}
17:16:39 [17:16:39.695]
INFO:
prover-client:proving-broker Proving Broker started
17:16:39 [17:16:39.696]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c76a8d342ceb5edb epochNumber=1
{"provingJobId":"1:BASE_PARITY:c76a8d342ceb5edb"}
17:16:39 [17:16:39.696]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3eafed5d92168778 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3eafed5d92168778"}
17:16:39 [17:16:39.696]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c76a8d342ceb5edb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c76a8d342ceb5edb"}
17:16:39 [17:16:39.697]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c76a8d342ceb5edb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c76a8d342ceb5edb"}
17:16:39 [17:16:39.697]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3eafed5d92168778 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3eafed5d92168778"}
17:16:39 [17:16:39.698]
INFO:
prover-client:proving-broker Proving Broker started
17:16:39 [17:16:39.699]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7edaea8fc2267e8d err=test error
{"provingJoId":"1:BASE_PARITY:7edaea8fc2267e8d"}
17:16:39 [17:16:39.700]
INFO:
prover-client:proving-broker Proving Broker started
17:16:39 [17:16:39.700]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f58a2a7906d5e73b not found
{"provingJobId":"1:BASE_PARITY:f58a2a7906d5e73b"}
17:16:39 [17:16:39.701]
INFO:
prover-client:proving-broker Proving Broker started
17:16:39 [17:16:39.702]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03e9a5ab99bab6aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:03e9a5ab99bab6aa"}
17:16:39 [17:16:39.703]
INFO:
prover-client:proving-broker Proving Broker started
17:16:39 [17:16:39.703]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a0f54b06023743e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a0f54b06023743e3"}
17:16:39 [17:16:39.804]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a0f54b06023743e3 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a0f54b06023743e3"}
17:16:39 [17:16:39.812]
INFO:
prover-client:proving-broker Proving Broker started
17:16:39 [17:16:39.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c12dd72d5908d84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c12dd72d5908d84"}
17:16:39 [17:16:39.914]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8c12dd72d5908d84 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8c12dd72d5908d84"}
17:16:39 [17:16:39.916]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:298a63172fef3d9a epochNumber=10
{"provingJobId":"1:BASE_PARITY:298a63172fef3d9a"}
17:16:39 [17:16:39.940]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:40 [17:16:40.043]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.044]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b69d6daf3248c3c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b69d6daf3248c3c"}
17:16:40 [17:16:40.199]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1b69d6daf3248c3c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1b69d6daf3248c3c"}
17:16:40 [17:16:40.247]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.248]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df2ad5f5fa7ea2e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df2ad5f5fa7ea2e6"}
17:16:40 [17:16:40.249]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:df2ad5f5fa7ea2e6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:df2ad5f5fa7ea2e6"}
17:16:40 [17:16:40.250]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9240c77702de656f epochNumber=1
{"provingJobId":"1:BASE_PARITY:9240c77702de656f"}
17:16:40 [17:16:40.251]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9240c77702de656f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:9240c77702de656f"}
17:16:40 [17:16:40.251]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9240c77702de656f type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:9240c77702de656f"}
17:16:40 [17:16:40.252]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:094f70146fef9483 epochNumber=1
{"provingJobId":"1:BASE_PARITY:094f70146fef9483"}
17:16:40 [17:16:40.253]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:094f70146fef9483 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:094f70146fef9483"}
17:16:40 [17:16:40.254]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dec96a90d5ffd0b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dec96a90d5ffd0b5"}
17:16:40 [17:16:40.255]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:dec96a90d5ffd0b5 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:dec96a90d5ffd0b5"}
17:16:40 [17:16:40.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b82fd3c528cf634 epochNumber=3
{"provingJobId":"1:BASE_PARITY:2b82fd3c528cf634"}
17:16:40 [17:16:40.279]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:40 [17:16:40.281]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:dec96a90d5ffd0b5 err=test error
{"provingJoId":"1:BASE_PARITY:dec96a90d5ffd0b5"}
17:16:40 [17:16:40.284]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.284]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:99618fb3a6121e9a settled=false
{"provingJobId":"1:BASE_PARITY:99618fb3a6121e9a","status":"pending"}
17:16:40 [17:16:40.284]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d85e2558f9c36b33 settled=false
{"provingJobId":"1:BASE_PARITY:d85e2558f9c36b33","status":"pending"}
17:16:40 [17:16:40.287]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.287]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c2a05a570d360ca7 settled=true
{"provingJobId":"1:BASE_PARITY:c2a05a570d360ca7","status":"fulfilled"}
17:16:40 [17:16:40.288]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:e8566d11d53b4581 settled=true
{"provingJobId":"2:BASE_PARITY:e8566d11d53b4581","status":"fulfilled"}
17:16:40 [17:16:40.289]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.290]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:692c769adaba8374 settled=true
{"provingJobId":"1:BASE_PARITY:692c769adaba8374","status":"fulfilled"}
17:16:40 [17:16:40.290]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:616b8c2682ba5795 settled=false
{"provingJobId":"1:BASE_PARITY:616b8c2682ba5795","status":"pending"}
17:16:40 [17:16:40.292]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.294]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e388e9941fdb4c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e388e9941fdb4c1"}
17:16:40 [17:16:40.296]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.297]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ac6b51d48b40332 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ac6b51d48b40332"}
17:16:40 [17:16:40.302]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:9ac6b51d48b40332: Error: db error: Error: db error
17:16:40 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:9ac6b51d48b40332"}
17:16:40 [17:16:40.305]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.306]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1594fcaf70d95a8c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1594fcaf70d95a8c"}
17:16:40 [17:16:40.308]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1594fcaf70d95a8c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1594fcaf70d95a8c"}
17:16:40 [17:16:40.308]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1594fcaf70d95a8c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1594fcaf70d95a8c"}
17:16:40 [17:16:40.310]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.310]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56688356cc9e0d97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56688356cc9e0d97"}
17:16:40 [17:16:40.310]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:56688356cc9e0d97 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:56688356cc9e0d97"}
17:16:40 [17:16:40.311]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:56688356cc9e0d97 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:56688356cc9e0d97"}
17:16:40 [17:16:40.312]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:56688356cc9e0d97: Error: db error
17:16:40 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:56688356cc9e0d97"}
17:16:40 [17:16:40.313]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.314]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c20771cce7a0c5c epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c20771cce7a0c5c"}
17:16:40 [17:16:40.314]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6c20771cce7a0c5c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6c20771cce7a0c5c"}
17:16:40 [17:16:40.314]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6c20771cce7a0c5c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6c20771cce7a0c5c"}
17:16:40 [17:16:40.315]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.316]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce8d0cbba9cfc9ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:ce8d0cbba9cfc9ad"}
17:16:40 [17:16:40.316]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ce8d0cbba9cfc9ad type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ce8d0cbba9cfc9ad"}
17:16:40 [17:16:40.316]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ce8d0cbba9cfc9ad type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ce8d0cbba9cfc9ad"}
17:16:40 [17:16:40.317]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:ce8d0cbba9cfc9ad jobErr=test error: Error: db error
17:16:40 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:ce8d0cbba9cfc9ad"}
17:16:40 [17:16:40.318]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.318]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:faf16bf8cabbf4b2 not found
{"provingJobId":"1:BASE_PARITY:faf16bf8cabbf4b2"}
17:16:40 [17:16:40.319]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.320]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:be26ec74a5217d77 err=test error
{"provingJoId":"1:BASE_PARITY:be26ec74a5217d77"}
17:16:40 [17:16:40.321]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.332]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ac9918b9168572eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:ac9918b9168572eb"}
17:16:40 [17:16:40.333]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ac9918b9168572eb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ac9918b9168572eb"}
17:16:40 [17:16:40.333]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ac9918b9168572eb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ac9918b9168572eb"}
17:16:40 [17:16:40.343]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:1ecba5200afb6043 epochNumber=2
{"provingJobId":"2:BASE_PARITY:1ecba5200afb6043"}
17:16:40 [17:16:40.344]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:1ecba5200afb6043 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:1ecba5200afb6043"}
17:16:40 [17:16:40.344]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:1ecba5200afb6043 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:1ecba5200afb6043"}
17:16:40 [17:16:40.405]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:59d9035d5626e327 epochNumber=3
{"provingJobId":"3:BASE_PARITY:59d9035d5626e327"}
17:16:40 [17:16:40.425]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:40 [17:16:40.467]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:65a61ec67377027e epochNumber=4
{"provingJobId":"4:BASE_PARITY:65a61ec67377027e"}
17:16:40 [17:16:40.477]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:40 [17:16:40.529]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:aad00b777bad8785 epochNumber=5
{"provingJobId":"5:BASE_PARITY:aad00b777bad8785"}
17:16:40 [17:16:40.530]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:40 [17:16:40.597]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.598]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36a90ad7c6a44453 epochNumber=42
{"provingJobId":"1:BASE_PARITY:36a90ad7c6a44453"}
17:16:40 [17:16:40.600]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testROoUMd/42 with map size 1073741824
17:16:40 [17:16:40.606]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.631]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:56e06be49f110497 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56e06be49f110497"}
17:16:40 [17:16:40.636]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.637]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f5042bd327956fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f5042bd327956fe"}
17:16:40 [17:16:40.638]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPCAjeQ/1 with map size 1073741824
17:16:40 [17:16:40.639]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.661]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ba0232dd512c4b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ba0232dd512c4b4"}
17:16:40 [17:16:40.679]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.680]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2d4a99708c06463 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d2d4a99708c06463"}
17:16:40 [17:16:40.682]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYd2ued/1 with map size 1073741824
17:16:40 [17:16:40.687]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.706]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d2d4a99708c06463 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d2d4a99708c06463"}
17:16:40 [17:16:40.710]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.711]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca0515158308f84f epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca0515158308f84f"}
17:16:40 [17:16:40.712]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMvXXgN/1 with map size 1073741824
17:16:40 [17:16:40.714]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.734]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ca0515158308f84f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ca0515158308f84f"}
17:16:40 [17:16:40.735]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ca0515158308f84f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ca0515158308f84f"}
17:16:40 [17:16:40.742]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ca0515158308f84f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ca0515158308f84f"}
17:16:40 [17:16:40.745]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.746]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:895d76877f93cffa epochNumber=1
{"provingJobId":"1:BASE_PARITY:895d76877f93cffa"}
17:16:40 [17:16:40.747]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsbx2GH/1 with map size 1073741824
17:16:40 [17:16:40.750]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.779]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.783]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a75af17fbb664f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a75af17fbb664f5"}
17:16:40 [17:16:40.784]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBZBjqz/1 with map size 1073741824
17:16:40 [17:16:40.787]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.810]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1a75af17fbb664f5
{"provingJobId":"1:BASE_PARITY:1a75af17fbb664f5"}
17:16:40 [17:16:40.810]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1a75af17fbb664f5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1a75af17fbb664f5"}
17:16:40 [17:16:40.810]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1a75af17fbb664f5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1a75af17fbb664f5"}
17:16:40 [17:16:40.820]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.821]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4476730ba90dc8b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4476730ba90dc8b9"}
17:16:40 [17:16:40.821]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaSJ9s7/1 with map size 1073741824
17:16:40 [17:16:40.823]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.841]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4476730ba90dc8b9
{"provingJobId":"1:BASE_PARITY:4476730ba90dc8b9"}
17:16:40 [17:16:40.842]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4476730ba90dc8b9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4476730ba90dc8b9"}
17:16:40 [17:16:40.850]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.851]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f21fd2bfffde66f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f21fd2bfffde66f9"}
17:16:40 [17:16:40.851]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdjxOE5/1 with map size 1073741824
17:16:40 [17:16:40.854]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.872]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f21fd2bfffde66f9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f21fd2bfffde66f9"}
17:16:40 [17:16:40.873]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f21fd2bfffde66f9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f21fd2bfffde66f9"}
17:16:40 [17:16:40.892]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.893]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17abedfe8820b60e epochNumber=1
{"provingJobId":"1:BASE_PARITY:17abedfe8820b60e"}
17:16:40 [17:16:40.894]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testElOHEH/1 with map size 1073741824
17:16:40 [17:16:40.903]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.918]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:17abedfe8820b60e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:17abedfe8820b60e"}
17:16:40 [17:16:40.919]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:17abedfe8820b60e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:17abedfe8820b60e"}
17:16:40 [17:16:40.926]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.927]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:443f2598847e8f66 epochNumber=0
{"provingJobId":"1:BASE_PARITY:443f2598847e8f66"}
17:16:40 [17:16:40.927]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testfolLrl/0 with map size 1073741824
17:16:40 [17:16:40.929]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.944]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:443f2598847e8f66 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:443f2598847e8f66"}
17:16:40 [17:16:40.945]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c91c82f174484baf epochNumber=0
{"provingJobId":"1:BASE_PARITY:c91c82f174484baf"}
17:16:40 [17:16:40.946]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:443f2598847e8f66 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:443f2598847e8f66"}
17:16:40 [17:16:40.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b8cfdc7698df9f1 epochNumber=0
{"provingJobId":"1:BASE_PARITY:8b8cfdc7698df9f1"}
17:16:40 [17:16:40.958]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.960]
INFO:
prover-client:proving-broker Proving Broker started
17:16:40 [17:16:40.961]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f16466379bcb344 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1f16466379bcb344"}
17:16:40 [17:16:40.961]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testve39Nw/2 with map size 1073741824
17:16:40 [17:16:40.963]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:40 [17:16:40.993]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3848e5d92e84d8d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3848e5d92e84d8d3"}
17:16:40 [17:16:40.994]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testve39Nw/1 with map size 1073741824
17:16:40 [17:16:40.996]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.019]
INFO:
prover-client:proving-broker Proving Broker started
17:16:41 [17:16:41.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16b3d256c44bd3be epochNumber=1
{"provingJobId":"1:BASE_PARITY:16b3d256c44bd3be"}
17:16:41 [17:16:41.021]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYEFvxZ/1 with map size 1073741824
17:16:41 [17:16:41.023]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.044]
INFO:
prover-client:proving-broker Proving Broker started
17:16:41 [17:16:41.045]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3fd02cf86b13372e epochNumber=1
{"provingJobId":"1:BASE_PARITY:3fd02cf86b13372e"}
17:16:41 [17:16:41.045]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAyP4JJ/1 with map size 1073741824
17:16:41 [17:16:41.047]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.064]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d5a95a57140ee03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d5a95a57140ee03"}
17:16:41 [17:16:41.069]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50304359213642db epochNumber=2
{"provingJobId":"1:BASE_PARITY:50304359213642db"}
17:16:41 [17:16:41.072]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAyP4JJ/2 with map size 1073741824
17:16:41 [17:16:41.081]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.103]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6802da157fc27b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6802da157fc27b8"}
17:16:41 [17:16:41.113]
INFO:
prover-client:proving-broker Proving Broker started
17:16:41 [17:16:41.114]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b997ea9488f0113d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b997ea9488f0113d"}
17:16:41 [17:16:41.115]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOKEx9q/1 with map size 1073741824
17:16:41 [17:16:41.117]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.134]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a56a501cc339502 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6a56a501cc339502"}
17:16:41 [17:16:41.140]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32840867f9b00225 epochNumber=2
{"provingJobId":"1:BASE_PARITY:32840867f9b00225"}
17:16:41 [17:16:41.140]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOKEx9q/2 with map size 1073741824
17:16:41 [17:16:41.142]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.157]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51231f78d2b5af54 epochNumber=1
{"provingJobId":"1:BASE_PARITY:51231f78d2b5af54"}
17:16:41 [17:16:41.164]
INFO:
prover-client:proving-broker Proving Broker started
17:16:41 [17:16:41.165]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a5b351cfc18357b epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a5b351cfc18357b"}
17:16:41 [17:16:41.165]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSiDtAV/1 with map size 1073741824
17:16:41 [17:16:41.167]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.193]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26f9c444faa65552 epochNumber=1
{"provingJobId":"1:BASE_PARITY:26f9c444faa65552"}
17:16:41 [17:16:41.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:850e697ab93b87ff epochNumber=2
{"provingJobId":"1:BASE_PARITY:850e697ab93b87ff"}
17:16:41 [17:16:41.201]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSiDtAV/2 with map size 1073741824
17:16:41 [17:16:41.204]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.224]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:629b9c8aaa654b97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:629b9c8aaa654b97"}
17:16:41 [17:16:41.234]
INFO:
prover-client:proving-broker Proving Broker started
17:16:41 [17:16:41.234]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c033b1ed242da09 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c033b1ed242da09"}
17:16:41 [17:16:41.235]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOW07zk/1 with map size 1073741824
17:16:41 [17:16:41.238]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8171db127dfcad28 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8171db127dfcad28"}
17:16:41 [17:16:41.264]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8c033b1ed242da09 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8c033b1ed242da09"}
17:16:41 [17:16:41.280]
INFO:
prover-client:proving-broker Proving Broker started
17:16:41 [17:16:41.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cacabc83d0e7cc15 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cacabc83d0e7cc15"}
17:16:41 [17:16:41.281]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdHrRsQ/1 with map size 1073741824
17:16:41 [17:16:41.283]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.303]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f536bade9eafdbd epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f536bade9eafdbd"}
17:16:41 [17:16:41.309]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cacabc83d0e7cc15 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:cacabc83d0e7cc15"}
17:16:41 [17:16:41.318]
INFO:
prover-client:proving-broker Proving Broker started
17:16:41 [17:16:41.319]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19b5b37c56a25757 epochNumber=1
{"provingJobId":"1:BASE_PARITY:19b5b37c56a25757"}
17:16:41 [17:16:41.320]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYSeLmq/1 with map size 1073741824
17:16:41 [17:16:41.324]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.341]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75ee076131041ece epochNumber=1
{"provingJobId":"1:BASE_PARITY:75ee076131041ece"}
17:16:41 [17:16:41.346]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:19b5b37c56a25757 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:19b5b37c56a25757"}
17:16:41 [17:16:41.351]
INFO:
prover-client:proving-broker Proving Broker started
17:16:41 [17:16:41.351]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5c6772ccf97ffff epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5c6772ccf97ffff"}
17:16:41 [17:16:41.352]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testE6vYIw/1 with map size 1073741824
17:16:41 [17:16:41.354]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.372]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b5c6772ccf97ffff
{"provingJobId":"1:BASE_PARITY:b5c6772ccf97ffff"}
17:16:41 [17:16:41.373]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b5c6772ccf97ffff type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b5c6772ccf97ffff"}
17:16:41 [17:16:41.382]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1fa5124b4f5f920c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1fa5124b4f5f920c"}
17:16:41 [17:16:41.392]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b5c6772ccf97ffff has already been completed
{"provingJobId":"1:BASE_PARITY:b5c6772ccf97ffff"}
17:16:41 [17:16:41.397]
INFO:
prover-client:proving-broker Proving Broker started
17:16:41 [17:16:41.398]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61e09bdd581d3e93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:61e09bdd581d3e93"}
17:16:41 [17:16:41.399]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlaeXEB/1 with map size 1073741824
17:16:41 [17:16:41.402]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:41 [17:16:41.421]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e9edec9e29bf586 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8e9edec9e29bf586"}
17:16:41 [17:16:41.422]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlaeXEB/2 with map size 1073741824
17:16:41 [17:16:41.425]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:42 [17:16:42.494]
INFO:
prover-client:proving-broker Proving Broker started
17:16:42 [17:16:42.500]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:61e09bdd581d3e93 settled=false
{"provingJobId":"1:BASE_PARITY:61e09bdd581d3e93","status":"pending"}
17:16:42 [17:16:42.503]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8e9edec9e29bf586 settled=false
{"provingJobId":"1:BASE_PARITY:8e9edec9e29bf586","status":"pending"}
17:16:42 [17:16:42.504]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:61e09bdd581d3e93 type=BASE_PARITY startedAt=1750094201442 older agent has taken job
{"provingJobId":"1:BASE_PARITY:61e09bdd581d3e93"}
17:16:42 [17:16:42.504]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:61e09bdd581d3e93 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:61e09bdd581d3e93"}
17:16:42 [17:16:42.508]
INFO:
prover-client:proving-broker Proving Broker started
17:16:42 [17:16:42.509]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:242f6381d798d4b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:242f6381d798d4b1"}
17:16:42 [17:16:42.509]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyhkQxy/1 with map size 1073741824
17:16:42 [17:16:42.511]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:42 [17:16:42.531]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a322d2d17c38ae6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0a322d2d17c38ae6"}
17:16:42 [17:16:42.532]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyhkQxy/2 with map size 1073741824
17:16:42 [17:16:42.534]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:43 [17:16:43.556]
INFO:
prover-client:proving-broker Proving Broker started
17:16:43 [17:16:43.559]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:242f6381d798d4b1 settled=false
{"provingJobId":"1:BASE_PARITY:242f6381d798d4b1","status":"pending"}
17:16:43 [17:16:43.560]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0a322d2d17c38ae6 settled=false
{"provingJobId":"1:BASE_PARITY:0a322d2d17c38ae6","status":"pending"}
17:16:43 [17:16:43.562]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:242f6381d798d4b1 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:242f6381d798d4b1"}
17:16:43 [17:16:43.567]
INFO:
prover-client:proving-broker Proving Broker started
17:16:43 [17:16:43.568]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51d8fd03e8711bdb epochNumber=1
{"provingJobId":"1:BASE_PARITY:51d8fd03e8711bdb"}
17:16:43 [17:16:43.569]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMXhnB8/1 with map size 1073741824
17:16:43 [17:16:43.571]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:43 [17:16:43.589]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b155560700928da9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b155560700928da9"}
17:16:43 [17:16:43.590]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMXhnB8/2 with map size 1073741824
17:16:43 [17:16:43.593]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:44 [17:16:44.614]
INFO:
prover-client:proving-broker Proving Broker started
17:16:44 [17:16:44.617]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:51d8fd03e8711bdb settled=false
{"provingJobId":"1:BASE_PARITY:51d8fd03e8711bdb","status":"pending"}
17:16:44 [17:16:44.618]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b155560700928da9 settled=false
{"provingJobId":"1:BASE_PARITY:b155560700928da9","status":"pending"}
17:16:44 [17:16:44.619]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:51d8fd03e8711bdb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:51d8fd03e8711bdb"}
17:16:44 [17:16:44.619]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:51d8fd03e8711bdb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:51d8fd03e8711bdb"}
17:16:44 [17:16:44.630]
INFO:
prover-client:proving-broker Proving Broker started
17:16:44 [17:16:44.631]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0061db55cdb476c epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0061db55cdb476c"}
17:16:44 [17:16:44.632]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFJhbz5/1 with map size 1073741824
17:16:44 [17:16:44.635]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:44 [17:16:44.656]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15c615b2b0b4c414 epochNumber=2
{"provingJobId":"1:BASE_PARITY:15c615b2b0b4c414"}
17:16:44 [17:16:44.658]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFJhbz5/2 with map size 1073741824
17:16:44 [17:16:44.661]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:44 [17:16:44.690]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e0061db55cdb476c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e0061db55cdb476c"}
17:16:44 [17:16:44.698]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:15c615b2b0b4c414 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:15c615b2b0b4c414"}
17:16:44 [17:16:44.709]
INFO:
prover-client:proving-broker Proving Broker started
17:16:44 [17:16:44.710]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b4048812db394c29 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b4048812db394c29"}
17:16:44 [17:16:44.711]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJJt8y3/1 with map size 1073741824
17:16:44 [17:16:44.712]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:44 [17:16:44.731]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78444bd79a64295a epochNumber=2
{"provingJobId":"1:BASE_PARITY:78444bd79a64295a"}
17:16:44 [17:16:44.732]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJJt8y3/2 with map size 1073741824
17:16:44 [17:16:44.734]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:44 [17:16:44.753]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b4048812db394c29 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b4048812db394c29"}
17:16:44 [17:16:44.753]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b4048812db394c29 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b4048812db394c29"}
17:16:44 [17:16:44.759]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:78444bd79a64295a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:78444bd79a64295a"}
17:16:44 [17:16:44.767]
INFO:
prover-client:proving-broker Proving Broker started
17:16:44 [17:16:44.769]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c7efe445848c73bb err=test error
{"provingJoId":"1:BASE_PARITY:c7efe445848c73bb"}
17:16:44 [17:16:44.773]
INFO:
prover-client:proving-broker Proving Broker started
17:16:44 [17:16:44.774]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:52f0b12fac56aa85 not found
{"provingJobId":"1:BASE_PARITY:52f0b12fac56aa85"}
17:16:44 [17:16:44.777]
INFO:
prover-client:proving-broker Proving Broker started
17:16:44 [17:16:44.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f8aaf6feb33b8e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f8aaf6feb33b8e2"}
17:16:44 [17:16:44.779]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testl9fhuh/1 with map size 1073741824
17:16:44 [17:16:44.782]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:44 [17:16:44.806]
INFO:
prover-client:proving-broker Proving Broker started
17:16:44 [17:16:44.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e1957a11c86b3fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e1957a11c86b3fb"}
17:16:44 [17:16:44.808]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqzFnWs/1 with map size 1073741824
17:16:44 [17:16:44.809]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:44 [17:16:44.933]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5e1957a11c86b3fb timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5e1957a11c86b3fb"}
17:16:44 [17:16:44.943]
INFO:
prover-client:proving-broker Proving Broker started
17:16:44 [17:16:44.943]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1897f3ae65628806 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1897f3ae65628806"}
17:16:44 [17:16:44.944]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMsNl3U/1 with map size 1073741824
17:16:44 [17:16:44.946]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.070]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1897f3ae65628806 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1897f3ae65628806"}
17:16:45 [17:16:45.072]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:424144160373b0e3 epochNumber=10
{"provingJobId":"1:BASE_PARITY:424144160373b0e3"}
17:16:45 [17:16:45.078]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testMsNl3U/10 with map size 1073741824
17:16:45 [17:16:45.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.122]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:45 [17:16:45.122]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:16:45 [17:16:45.241]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.242]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:128d191f033cb9b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:128d191f033cb9b3"}
17:16:45 [17:16:45.244]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEBPDla/1 with map size 1073741824
17:16:45 [17:16:45.245]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.425]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:128d191f033cb9b3 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:128d191f033cb9b3"}
17:16:45 [17:16:45.472]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.474]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ad9c5294207506a epochNumber=1
{"provingJobId":"1:BASE_PARITY:1ad9c5294207506a"}
17:16:45 [17:16:45.475]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEMFN9n/1 with map size 1073741824
17:16:45 [17:16:45.477]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.496]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1ad9c5294207506a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1ad9c5294207506a"}
17:16:45 [17:16:45.500]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.501]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a08ed83e5370aa2f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a08ed83e5370aa2f"}
17:16:45 [17:16:45.502]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbEsEer/1 with map size 1073741824
17:16:45 [17:16:45.505]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.523]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a08ed83e5370aa2f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a08ed83e5370aa2f"}
17:16:45 [17:16:45.524]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a08ed83e5370aa2f type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:a08ed83e5370aa2f"}
17:16:45 [17:16:45.533]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:875977f50e2d4a8a epochNumber=1
{"provingJobId":"1:BASE_PARITY:875977f50e2d4a8a"}
17:16:45 [17:16:45.536]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testY1cJpl/1 with map size 1073741824
17:16:45 [17:16:45.538]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.555]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:875977f50e2d4a8a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:875977f50e2d4a8a"}
17:16:45 [17:16:45.564]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.565]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:33331e587b55dbf5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:33331e587b55dbf5"}
17:16:45 [17:16:45.565]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpvuuuT/1 with map size 1073741824
17:16:45 [17:16:45.567]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.593]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:33331e587b55dbf5 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:33331e587b55dbf5"}
17:16:45 [17:16:45.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a229278a6e3017b0 epochNumber=3
{"provingJobId":"1:BASE_PARITY:a229278a6e3017b0"}
17:16:45 [17:16:45.595]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testpvuuuT/3 with map size 1073741824
17:16:45 [17:16:45.599]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.640]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:45 [17:16:45.641]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:16:45 [17:16:45.645]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:33331e587b55dbf5 err=test error
{"provingJoId":"1:BASE_PARITY:33331e587b55dbf5"}
17:16:45 [17:16:45.650]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpuf4xG/1 with map size 1073741824
17:16:45 [17:16:45.652]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.675]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpuf4xG/2 with map size 1073741824
17:16:45 [17:16:45.683]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.705]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.707]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8e955583d6de4c46 settled=false
{"provingJobId":"1:BASE_PARITY:8e955583d6de4c46","status":"pending"}
17:16:45 [17:16:45.708]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9071a1d7831083ce settled=false
{"provingJobId":"1:BASE_PARITY:9071a1d7831083ce","status":"pending"}
17:16:45 [17:16:45.715]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFBeYA9/1 with map size 1073741824
17:16:45 [17:16:45.719]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.740]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFBeYA9/2 with map size 1073741824
17:16:45 [17:16:45.743]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.783]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.786]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dc99e3a51c00a6e7 settled=true
{"provingJobId":"1:BASE_PARITY:dc99e3a51c00a6e7","status":"fulfilled"}
17:16:45 [17:16:45.789]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:d3519b6a8e31c21d settled=true
{"provingJobId":"2:BASE_PARITY:d3519b6a8e31c21d","status":"fulfilled"}
17:16:45 [17:16:45.799]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcB0Npa/1 with map size 1073741824
17:16:45 [17:16:45.801]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.840]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcB0Npa/2 with map size 1073741824
17:16:45 [17:16:45.844]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.865]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.867]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2db1ddf24ff77ba5 settled=true
{"provingJobId":"1:BASE_PARITY:2db1ddf24ff77ba5","status":"fulfilled"}
17:16:45 [17:16:45.868]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:85b83fc1fb2c53e5 settled=false
{"provingJobId":"1:BASE_PARITY:85b83fc1fb2c53e5","status":"pending"}
17:16:45 [17:16:45.874]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.875]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0bc658f6c503087 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0bc658f6c503087"}
17:16:45 [17:16:45.877]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testy942Ck/1 with map size 1073741824
17:16:45 [17:16:45.881]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.904]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.904]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e67bfc29ae2038d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e67bfc29ae2038d3"}
17:16:45 [17:16:45.905]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:e67bfc29ae2038d3: Error: db error: Error: db error
17:16:45 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:e67bfc29ae2038d3"}
17:16:45 [17:16:45.910]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.911]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40aa7478c29ae746 epochNumber=1
{"provingJobId":"1:BASE_PARITY:40aa7478c29ae746"}
17:16:45 [17:16:45.911]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTVnFII/1 with map size 1073741824
17:16:45 [17:16:45.913]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.931]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:40aa7478c29ae746 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:40aa7478c29ae746"}
17:16:45 [17:16:45.932]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:40aa7478c29ae746 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:40aa7478c29ae746"}
17:16:45 [17:16:45.943]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17c199386a5591d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17c199386a5591d0"}
17:16:45 [17:16:45.944]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3guTFD/1 with map size 1073741824
17:16:45 [17:16:45.946]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.965]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:17c199386a5591d0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:17c199386a5591d0"}
17:16:45 [17:16:45.965]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:17c199386a5591d0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:17c199386a5591d0"}
17:16:45 [17:16:45.966]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:17c199386a5591d0: Error: db error
17:16:45 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:17c199386a5591d0"}
17:16:45 [17:16:45.974]
INFO:
prover-client:proving-broker Proving Broker started
17:16:45 [17:16:45.974]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b8182a96fb91067 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b8182a96fb91067"}
17:16:45 [17:16:45.975]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOpF1G0/1 with map size 1073741824
17:16:45 [17:16:45.979]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:45 [17:16:45.998]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1b8182a96fb91067 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1b8182a96fb91067"}
17:16:45 [17:16:45.998]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1b8182a96fb91067 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1b8182a96fb91067"}
17:16:46 [17:16:46.012]
INFO:
prover-client:proving-broker Proving Broker started
17:16:46 [17:16:46.013]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8125ec452a11795e epochNumber=1
{"provingJobId":"1:BASE_PARITY:8125ec452a11795e"}
17:16:46 [17:16:46.014]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLzN1EJ/1 with map size 1073741824
17:16:46 [17:16:46.018]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:46 [17:16:46.038]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8125ec452a11795e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8125ec452a11795e"}
17:16:46 [17:16:46.039]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8125ec452a11795e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8125ec452a11795e"}
17:16:46 [17:16:46.040]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:8125ec452a11795e jobErr=test error: Error: db error
17:16:46 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:8125ec452a11795e"}
17:16:46 [17:16:46.046]
INFO:
prover-client:proving-broker Proving Broker started
17:16:46 [17:16:46.046]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:00fffd1b552a389f not found
{"provingJobId":"1:BASE_PARITY:00fffd1b552a389f"}
17:16:46 [17:16:46.054]
INFO:
prover-client:proving-broker Proving Broker started
17:16:46 [17:16:46.055]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:20cab7cf59f48ee4 err=test error
{"provingJoId":"1:BASE_PARITY:20cab7cf59f48ee4"}
17:16:46 [17:16:46.059]
INFO:
prover-client:proving-broker Proving Broker started
17:16:46 [17:16:46.070]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b49cdb52f3ebab1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b49cdb52f3ebab1"}
17:16:46 [17:16:46.071]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testa9RgYF/1 with map size 1073741824
17:16:46 [17:16:46.075]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:46 [17:16:46.099]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0b49cdb52f3ebab1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0b49cdb52f3ebab1"}
17:16:46 [17:16:46.100]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0b49cdb52f3ebab1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0b49cdb52f3ebab1"}
17:16:46 [17:16:46.117]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:363ae6bb51c5c335 epochNumber=2
{"provingJobId":"2:BASE_PARITY:363ae6bb51c5c335"}
17:16:46 [17:16:46.119]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testa9RgYF/2 with map size 1073741824
17:16:46 [17:16:46.127]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:46 [17:16:46.154]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:363ae6bb51c5c335 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:363ae6bb51c5c335"}
17:16:46 [17:16:46.155]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:363ae6bb51c5c335 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:363ae6bb51c5c335"}
17:16:46 [17:16:46.223]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:a1fc42031b611c70 epochNumber=3
{"provingJobId":"3:BASE_PARITY:a1fc42031b611c70"}
17:16:46 [17:16:46.225]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testa9RgYF/3 with map size 1073741824
17:16:46 [17:16:46.232]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:46 [17:16:46.263]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:46 [17:16:46.264]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:16:46 [17:16:46.312]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:518adfdd8485c266 epochNumber=4
{"provingJobId":"4:BASE_PARITY:518adfdd8485c266"}
17:16:46 [17:16:46.314]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testa9RgYF/4 with map size 1073741824
17:16:46 [17:16:46.317]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:46 [17:16:46.344]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:46 [17:16:46.345]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
17:16:46 [17:16:46.399]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:8f4b56f674c17f24 epochNumber=5
{"provingJobId":"5:BASE_PARITY:8f4b56f674c17f24"}
17:16:46 [17:16:46.400]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testa9RgYF/5 with map size 1073741824
17:16:46 [17:16:46.405]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:16:46 [17:16:46.451]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:16:46 [17:16:46.452]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
17:16:46
PASS src/proving_broker/proving_broker.test.ts (
13.646 s)
17:16:46 ProvingBroker
17:16:46 Producer API
17:16:46
✓ refuses stale jobs (57 ms)
17:16:46
✓ enqueues jobs (5 ms)
17:16:46
✓ ignores duplicate jobs (5 ms)
17:16:46
✓ reports correct status when enqueuing repeat jobs (5 ms)
17:16:46
✓ throws an error in case of duplicate job IDs (32 ms)
17:16:46
✓ returns not-found status for non-existing jobs (2 ms)
17:16:46
✓ cancels jobs in queue (4 ms)
17:16:46
✓ cancels jobs in-progress (3 ms)
17:16:46
✓ returns job result if successful (3 ms)
17:16:46
✓ returns job error if failed (2 ms)
17:16:46
✓ correctly returns job status for concurrent writes (3 ms)
17:16:46
✓ refuses stale jobs (47 ms)
17:16:46
✓ enqueues jobs (34 ms)
17:16:46
✓ ignores duplicate jobs (40 ms)
17:16:46
✓ reports correct status when enqueuing repeat jobs (34 ms)
17:16:46
✓ throws an error in case of duplicate job IDs (31 ms)
17:16:46
✓ returns not-found status for non-existing jobs (5 ms)
17:16:46
✓ cancels jobs in queue (37 ms)
17:16:46
✓ cancels jobs in-progress (30 ms)
17:16:46
✓ returns job result if successful (39 ms)
17:16:46
✓ returns job error if failed (36 ms)
17:16:46
✓ correctly returns job status for concurrent writes (32 ms)
17:16:46 Consumer API
17:16:46
✓ returns undefined if no jobs are available (1 ms)
17:16:46
✓ returns jobs in priority order (2 ms)
17:16:46
✓ returns undefined if no jobs are available for the given allowList (1 ms)
17:16:46
✓ returns a job if it is in the allowList (2 ms)
17:16:46
✓ returns the most important job if it is in the allowList (2 ms)
17:16:46
✓ returns any job if filter is empty (2 ms)
17:16:46
✓ returns a new job when reporting job success (2 ms)
17:16:46
✓ returns a new job when reporting permanent error (2 ms)
17:16:46
✓ returns a new job when reporting retry-able error (2 ms)
17:16:46
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
17:16:46
✓ returns a new job if job is already in progress elsewhere (1057 ms)
17:16:46
✓ avoids sending the same job to a new agent after a restart (1004 ms)
17:16:46
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
17:16:46
✓ tracks job result if in progress (3 ms)
17:16:46
✓ tracks job result even if job is in queue (3 ms)
17:16:46
✓ ignores reported job error if unknown job (1 ms)
17:16:46
✓ ignores job result if unknown job (2 ms)
17:16:46
✓ returns undefined if no jobs are available (2 ms)
17:16:46
✓ returns jobs in priority order (56 ms)
17:16:46
✓ returns undefined if no jobs are available for the given allowList (26 ms)
17:16:46
✓ returns a job if it is in the allowList (67 ms)
17:16:46
✓ returns the most important job if it is in the allowList (52 ms)
17:16:46
✓ returns any job if filter is empty (69 ms)
17:16:46
✓ returns a new job when reporting job success (45 ms)
17:16:46
✓ returns a new job when reporting permanent error (38 ms)
17:16:46
✓ returns a new job when reporting retry-able error (31 ms)
17:16:46
✓ returns a new job when reporting progress if current one is cancelled (46 ms)
17:16:46
✓ returns a new job if job is already in progress elsewhere (1111 ms)
17:16:46
✓ avoids sending the same job to a new agent after a restart (1058 ms)
17:16:46
✓ avoids sending a completed job to a new agent after a restart (1063 ms)
17:16:46
✓ tracks job result if in progress (79 ms)
17:16:46
✓ tracks job result even if job is in queue (59 ms)
17:16:46
✓ ignores reported job error if unknown job (3 ms)
17:16:46
✓ ignores job result if unknown job (5 ms)
17:16:46 Timeouts
17:16:46
✓ tracks in progress jobs (1 ms)
17:16:46
✓ re-enqueues jobs that time out (108 ms)
17:16:46
✓ cancel stale jobs that time out (231 ms)
17:16:46
✓ keeps the jobs in progress while it is alive (203 ms)
17:16:46
✓ tracks in progress jobs (28 ms)
17:16:46
✓ re-enqueues jobs that time out (135 ms)
17:16:46
✓ cancel stale jobs that time out (299 ms)
17:16:46
✓ keeps the jobs in progress while it is alive (230 ms)
17:16:46 Retries
17:16:46
✓ retries jobs (2 ms)
17:16:46
✓ retries up to a maximum number of times (2 ms)
17:16:46
✓ passing retry=false does not retry (2 ms)
17:16:46
✓ does not retry if job is stale (28 ms)
17:16:46
✓ retries jobs (28 ms)
17:16:46
✓ retries up to a maximum number of times (32 ms)
17:16:46
✓ passing retry=false does not retry (30 ms)
17:16:46
✓ does not retry if job is stale (84 ms)
17:16:46 Database management
17:16:46
✓ re-enqueues proof requests on start (4 ms)
17:16:46
✓ restores proof results on start (2 ms)
17:16:46
✓ only re-enqueues unfinished jobs (2 ms)
17:16:46
✓ saves job when enqueued (4 ms)
17:16:46
✓ does not retain job if database fails to save (9 ms)
17:16:46
✓ saves job result (5 ms)
17:16:46
✓ saves result even if database fails to save (3 ms)
17:16:46
✓ saves job error (2 ms)
17:16:46
✓ saves job error even if database fails to save (2 ms)
17:16:46
✓ does not save job result if job is unknown (2 ms)
17:16:46
✓ does not save job error if job is unknown (1 ms)
17:16:46
✓ cleans up old jobs periodically (265 ms)
17:16:46
✓ re-enqueues proof requests on start (65 ms)
17:16:46
✓ restores proof results on start (81 ms)
17:16:46
✓ only re-enqueues unfinished jobs (78 ms)
17:16:46
✓ saves job when enqueued (30 ms)
17:16:46
✓ does not retain job if database fails to save (4 ms)
17:16:46
✓ saves job result (33 ms)
17:16:46
✓ saves result even if database fails to save (32 ms)
17:16:46
✓ saves job error (37 ms)
17:16:46
✓ saves job error even if database fails to save (33 ms)
17:16:46
✓ does not save job result if job is unknown (6 ms)
17:16:46
✓ does not save job error if job is unknown (8 ms)
17:16:46
✓ cleans up old jobs periodically (434 ms)
17:16:46
17:16:46
Test Suites: 1 passed, 1 total
17:16:46
Tests: 96 passed, 96 total
17:16:46
Snapshots: 0 total
17:16:46
Time: 13.765 s
17:16:46
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
17:16:46
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?