Parent Log:
http://ci.aztec-labs.com/ddf305e3c7df868f
Command: 29957fbe93665d48 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/4608046e462faf9ff3fbee265ba34dbecb5d0b31
Env: REF_NAME=gh-readonly-queue/next/pr-15169-2868628b6ae039db35b4267ab3975abc64a5d9b4 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 23 13:07:40 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15169_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
13:07:45 [13:07:45.149]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.156]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2fa03e1cfec75de1 epochNumber=42
{"provingJobId":"1:BASE_PARITY:2fa03e1cfec75de1"}
13:07:45 [13:07:45.159]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:5994a4bc3f359e3f epochNumber=1
{"provingJobId":"1:BASE_PARITY:5994a4bc3f359e3f"}
13:07:45 [13:07:45.190]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.191]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0364eec8360eb861 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0364eec8360eb861"}
13:07:45 [13:07:45.192]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca347d9a2368498a epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca347d9a2368498a"}
13:07:45 [13:07:45.194]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.195]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c62f4cd3b520c01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c62f4cd3b520c01"}
13:07:45 [13:07:45.196]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1c62f4cd3b520c01 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1c62f4cd3b520c01"}
13:07:45 [13:07:45.198]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.199]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f8cdf4461d624c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f8cdf4461d624c3"}
13:07:45 [13:07:45.200]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8f8cdf4461d624c3 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8f8cdf4461d624c3"}
13:07:45 [13:07:45.201]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8f8cdf4461d624c3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8f8cdf4461d624c3"}
13:07:45 [13:07:45.201]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8f8cdf4461d624c3 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8f8cdf4461d624c3"}
13:07:45 [13:07:45.203]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.203]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:946f5c1c88633cb4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:946f5c1c88633cb4"}
13:07:45 [13:07:45.228]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.229]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.230]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6eae4c597f80b38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6eae4c597f80b38"}
13:07:45 [13:07:45.230]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a6eae4c597f80b38
{"provingJobId":"1:BASE_PARITY:a6eae4c597f80b38"}
13:07:45 [13:07:45.231]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a6eae4c597f80b38 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a6eae4c597f80b38"}
13:07:45 [13:07:45.231]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a6eae4c597f80b38 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a6eae4c597f80b38"}
13:07:45 [13:07:45.233]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.233]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66390f65b4aef7a9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66390f65b4aef7a9"}
13:07:45 [13:07:45.234]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:66390f65b4aef7a9
{"provingJobId":"1:BASE_PARITY:66390f65b4aef7a9"}
13:07:45 [13:07:45.234]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:66390f65b4aef7a9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:66390f65b4aef7a9"}
13:07:45 [13:07:45.243]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.243]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a566632908818e79 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a566632908818e79"}
13:07:45 [13:07:45.244]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a566632908818e79 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a566632908818e79"}
13:07:45 [13:07:45.244]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a566632908818e79 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a566632908818e79"}
13:07:45 [13:07:45.246]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.247]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:edc545303ad7977a epochNumber=1
{"provingJobId":"1:BASE_PARITY:edc545303ad7977a"}
13:07:45 [13:07:45.247]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:edc545303ad7977a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:edc545303ad7977a"}
13:07:45 [13:07:45.247]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:edc545303ad7977a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:edc545303ad7977a"}
13:07:45 [13:07:45.249]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.249]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:532b391d3ccd8dac epochNumber=0
{"provingJobId":"1:BASE_PARITY:532b391d3ccd8dac"}
13:07:45 [13:07:45.249]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:532b391d3ccd8dac epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:532b391d3ccd8dac"}
13:07:45 [13:07:45.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2db6a1193a5a6be epochNumber=0
{"provingJobId":"1:BASE_PARITY:f2db6a1193a5a6be"}
13:07:45 [13:07:45.250]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:532b391d3ccd8dac epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:532b391d3ccd8dac"}
13:07:45 [13:07:45.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87ac562b883c7fa7 epochNumber=0
{"provingJobId":"1:BASE_PARITY:87ac562b883c7fa7"}
13:07:45 [13:07:45.251]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.253]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a62f6fdb3a36786 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3a62f6fdb3a36786"}
13:07:45 [13:07:45.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6eab64d69e3af08d epochNumber=1
{"provingJobId":"1:BASE_PARITY:6eab64d69e3af08d"}
13:07:45 [13:07:45.255]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1feb24ba95bcfca2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1feb24ba95bcfca2"}
13:07:45 [13:07:45.256]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.257]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c7361c20ac9e615f epochNumber=1
{"provingJobId":"1:BASE_PARITY:c7361c20ac9e615f"}
13:07:45 [13:07:45.257]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d34ad79a539be796 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d34ad79a539be796"}
13:07:45 [13:07:45.257]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91a8bc8f52c6fc9d epochNumber=2
{"provingJobId":"1:BASE_PARITY:91a8bc8f52c6fc9d"}
13:07:45 [13:07:45.257]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc7a1f64b5d1b6c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc7a1f64b5d1b6c8"}
13:07:45 [13:07:45.258]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.259]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08b1f03d7686353f epochNumber=1
{"provingJobId":"1:BASE_PARITY:08b1f03d7686353f"}
13:07:45 [13:07:45.259]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8036f9af148abcae epochNumber=1
{"provingJobId":"1:BASE_PARITY:8036f9af148abcae"}
13:07:45 [13:07:45.260]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef66c1239c49ca0d epochNumber=2
{"provingJobId":"1:BASE_PARITY:ef66c1239c49ca0d"}
13:07:45 [13:07:45.260]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b31f5ee6ea6151d8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b31f5ee6ea6151d8"}
13:07:45 [13:07:45.261]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6467f6a4b5ba2e78 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6467f6a4b5ba2e78"}
13:07:45 [13:07:45.262]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2f73aa5cd94f870 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2f73aa5cd94f870"}
13:07:45 [13:07:45.262]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a9f62634843c1bf epochNumber=2
{"provingJobId":"1:BASE_PARITY:5a9f62634843c1bf"}
13:07:45 [13:07:45.262]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:07a6fd9fa2a8e2c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:07a6fd9fa2a8e2c8"}
13:07:45 [13:07:45.263]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.263]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70bebe6192f8262d epochNumber=1
{"provingJobId":"1:BASE_PARITY:70bebe6192f8262d"}
13:07:45 [13:07:45.264]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2406230d3e42c5d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2406230d3e42c5d6"}
13:07:45 [13:07:45.264]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:70bebe6192f8262d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:70bebe6192f8262d"}
13:07:45 [13:07:45.265]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05866baeab7b386d epochNumber=1
{"provingJobId":"1:BASE_PARITY:05866baeab7b386d"}
13:07:45 [13:07:45.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8fca9089a4189020 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8fca9089a4189020"}
13:07:45 [13:07:45.266]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:05866baeab7b386d type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:05866baeab7b386d"}
13:07:45 [13:07:45.267]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eeb30bfabbd1ce7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:eeb30bfabbd1ce7d"}
13:07:45 [13:07:45.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88d9520feb1d4a56 epochNumber=1
{"provingJobId":"1:BASE_PARITY:88d9520feb1d4a56"}
13:07:45 [13:07:45.268]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:eeb30bfabbd1ce7d type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:eeb30bfabbd1ce7d"}
13:07:45 [13:07:45.269]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.270]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9fa19d496e2dfa76 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9fa19d496e2dfa76"}
13:07:45 [13:07:45.270]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9fa19d496e2dfa76
{"provingJobId":"1:BASE_PARITY:9fa19d496e2dfa76"}
13:07:45 [13:07:45.271]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9fa19d496e2dfa76 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:9fa19d496e2dfa76"}
13:07:45 [13:07:45.271]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:543926d44b8dad38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:543926d44b8dad38"}
13:07:45 [13:07:45.272]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9fa19d496e2dfa76 has already been completed
{"provingJobId":"1:BASE_PARITY:9fa19d496e2dfa76"}
13:07:45 [13:07:45.273]
INFO:
prover-client:proving-broker Proving Broker started
13:07:45 [13:07:45.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20d8f19972c5b58d epochNumber=1
{"provingJobId":"1:BASE_PARITY:20d8f19972c5b58d"}
13:07:45 [13:07:45.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e99f4348ce08d81 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2e99f4348ce08d81"}
13:07:46 [13:07:46.326]
INFO:
prover-client:proving-broker Proving Broker started
13:07:46 [13:07:46.326]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:20d8f19972c5b58d settled=false
{"provingJobId":"1:BASE_PARITY:20d8f19972c5b58d","status":"pending"}
13:07:46 [13:07:46.327]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2e99f4348ce08d81 settled=false
{"provingJobId":"1:BASE_PARITY:2e99f4348ce08d81","status":"pending"}
13:07:46 [13:07:46.328]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:20d8f19972c5b58d type=BASE_PARITY startedAt=1750684065274 older agent has taken job
{"provingJobId":"1:BASE_PARITY:20d8f19972c5b58d"}
13:07:46 [13:07:46.328]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:20d8f19972c5b58d type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:20d8f19972c5b58d"}
13:07:46 [13:07:46.329]
INFO:
prover-client:proving-broker Proving Broker started
13:07:46 [13:07:46.330]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31ee168b84af77f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:31ee168b84af77f8"}
13:07:46 [13:07:46.330]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c7cae453173f666 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6c7cae453173f666"}
13:07:47 [13:07:47.331]
INFO:
prover-client:proving-broker Proving Broker started
13:07:47 [13:07:47.332]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:31ee168b84af77f8 settled=false
{"provingJobId":"1:BASE_PARITY:31ee168b84af77f8","status":"pending"}
13:07:47 [13:07:47.332]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6c7cae453173f666 settled=false
{"provingJobId":"1:BASE_PARITY:6c7cae453173f666","status":"pending"}
13:07:47 [13:07:47.333]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:31ee168b84af77f8 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:31ee168b84af77f8"}
13:07:47 [13:07:47.334]
INFO:
prover-client:proving-broker Proving Broker started
13:07:47 [13:07:47.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c702c9b23ae221c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c702c9b23ae221c1"}
13:07:47 [13:07:47.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b39310f57e7a89a epochNumber=2
{"provingJobId":"1:BASE_PARITY:2b39310f57e7a89a"}
13:07:48 [13:07:48.336]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.337]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c702c9b23ae221c1 settled=false
{"provingJobId":"1:BASE_PARITY:c702c9b23ae221c1","status":"pending"}
13:07:48 [13:07:48.337]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2b39310f57e7a89a settled=false
{"provingJobId":"1:BASE_PARITY:2b39310f57e7a89a","status":"pending"}
13:07:48 [13:07:48.338]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c702c9b23ae221c1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c702c9b23ae221c1"}
13:07:48 [13:07:48.338]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c702c9b23ae221c1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c702c9b23ae221c1"}
13:07:48 [13:07:48.340]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.340]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88e038fa770a721f epochNumber=1
{"provingJobId":"1:BASE_PARITY:88e038fa770a721f"}
13:07:48 [13:07:48.340]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2b9a67ec32bc87f epochNumber=2
{"provingJobId":"1:BASE_PARITY:f2b9a67ec32bc87f"}
13:07:48 [13:07:48.341]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:88e038fa770a721f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:88e038fa770a721f"}
13:07:48 [13:07:48.342]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f2b9a67ec32bc87f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f2b9a67ec32bc87f"}
13:07:48 [13:07:48.343]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.343]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a6f9760908c519a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6a6f9760908c519a"}
13:07:48 [13:07:48.343]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98793baa4081c82d epochNumber=2
{"provingJobId":"1:BASE_PARITY:98793baa4081c82d"}
13:07:48 [13:07:48.344]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6a6f9760908c519a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6a6f9760908c519a"}
13:07:48 [13:07:48.344]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6a6f9760908c519a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6a6f9760908c519a"}
13:07:48 [13:07:48.344]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:98793baa4081c82d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:98793baa4081c82d"}
13:07:48 [13:07:48.345]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.346]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:73a35359ed6ea266 err=test error
{"provingJoId":"1:BASE_PARITY:73a35359ed6ea266"}
13:07:48 [13:07:48.347]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.347]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3f69bd1650ddbf26 not found
{"provingJobId":"1:BASE_PARITY:3f69bd1650ddbf26"}
13:07:48 [13:07:48.349]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.349]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1dc25be8d4499caf epochNumber=1
{"provingJobId":"1:BASE_PARITY:1dc25be8d4499caf"}
13:07:48 [13:07:48.350]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.351]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:942a7cbc038fbd52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:942a7cbc038fbd52"}
13:07:48 [13:07:48.454]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:942a7cbc038fbd52 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:942a7cbc038fbd52"}
13:07:48 [13:07:48.462]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.465]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af34691c9e7112c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af34691c9e7112c8"}
13:07:48 [13:07:48.566]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:af34691c9e7112c8 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:af34691c9e7112c8"}
13:07:48 [13:07:48.568]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:362d106108af09fe epochNumber=10
{"provingJobId":"1:BASE_PARITY:362d106108af09fe"}
13:07:48 [13:07:48.593]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:48 [13:07:48.695]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.696]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05e7468273fa9137 epochNumber=1
{"provingJobId":"1:BASE_PARITY:05e7468273fa9137"}
13:07:48 [13:07:48.849]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:05e7468273fa9137 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:05e7468273fa9137"}
13:07:48 [13:07:48.900]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.902]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bbb0c85c6f3c9a7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:bbb0c85c6f3c9a7d"}
13:07:48 [13:07:48.903]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bbb0c85c6f3c9a7d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:bbb0c85c6f3c9a7d"}
13:07:48 [13:07:48.904]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.905]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3f330887556f842 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3f330887556f842"}
13:07:48 [13:07:48.905]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d3f330887556f842 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d3f330887556f842"}
13:07:48 [13:07:48.906]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d3f330887556f842 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:d3f330887556f842"}
13:07:48 [13:07:48.908]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.908]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93a4116afb2898da epochNumber=1
{"provingJobId":"1:BASE_PARITY:93a4116afb2898da"}
13:07:48 [13:07:48.909]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:93a4116afb2898da type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:93a4116afb2898da"}
13:07:48 [13:07:48.910]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.910]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3990826d4df9bf7f epochNumber=1
{"provingJobId":"1:BASE_PARITY:3990826d4df9bf7f"}
13:07:48 [13:07:48.911]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3990826d4df9bf7f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3990826d4df9bf7f"}
13:07:48 [13:07:48.912]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:957ab9761f68b0f2 epochNumber=3
{"provingJobId":"1:BASE_PARITY:957ab9761f68b0f2"}
13:07:48 [13:07:48.935]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:48 [13:07:48.937]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3990826d4df9bf7f err=test error
{"provingJoId":"1:BASE_PARITY:3990826d4df9bf7f"}
13:07:48 [13:07:48.939]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.939]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2c934ca0a7e48804 settled=false
{"provingJobId":"1:BASE_PARITY:2c934ca0a7e48804","status":"pending"}
13:07:48 [13:07:48.939]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e185d2f1e47dcf66 settled=false
{"provingJobId":"1:BASE_PARITY:e185d2f1e47dcf66","status":"pending"}
13:07:48 [13:07:48.941]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.942]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e6ebb298a830d3b3 settled=true
{"provingJobId":"1:BASE_PARITY:e6ebb298a830d3b3","status":"fulfilled"}
13:07:48 [13:07:48.942]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:e0313d2188e1b11c settled=true
{"provingJobId":"2:BASE_PARITY:e0313d2188e1b11c","status":"fulfilled"}
13:07:48 [13:07:48.944]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.945]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f53ff82f1990efe8 settled=true
{"provingJobId":"1:BASE_PARITY:f53ff82f1990efe8","status":"fulfilled"}
13:07:48 [13:07:48.945]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c6c1cf15a512c8b1 settled=false
{"provingJobId":"1:BASE_PARITY:c6c1cf15a512c8b1","status":"pending"}
13:07:48 [13:07:48.955]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.958]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12b23b7d18ca0c1d epochNumber=1
{"provingJobId":"1:BASE_PARITY:12b23b7d18ca0c1d"}
13:07:48 [13:07:48.960]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.961]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ecd1b86c61a80b93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ecd1b86c61a80b93"}
13:07:48 [13:07:48.963]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:ecd1b86c61a80b93: Error: db error: Error: db error
13:07:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:ecd1b86c61a80b93"}
13:07:48 [13:07:48.964]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.965]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17f642bf7891fbc0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17f642bf7891fbc0"}
13:07:48 [13:07:48.966]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:17f642bf7891fbc0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:17f642bf7891fbc0"}
13:07:48 [13:07:48.967]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:17f642bf7891fbc0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:17f642bf7891fbc0"}
13:07:48 [13:07:48.968]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.969]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7873f170e8b024d epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7873f170e8b024d"}
13:07:48 [13:07:48.969]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e7873f170e8b024d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e7873f170e8b024d"}
13:07:48 [13:07:48.969]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e7873f170e8b024d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e7873f170e8b024d"}
13:07:48 [13:07:48.970]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:e7873f170e8b024d: Error: db error
13:07:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:e7873f170e8b024d"}
13:07:48 [13:07:48.971]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3fcd2ef7246a50ab epochNumber=1
{"provingJobId":"1:BASE_PARITY:3fcd2ef7246a50ab"}
13:07:48 [13:07:48.972]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3fcd2ef7246a50ab type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3fcd2ef7246a50ab"}
13:07:48 [13:07:48.972]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3fcd2ef7246a50ab type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3fcd2ef7246a50ab"}
13:07:48 [13:07:48.973]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.973]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a5b8907a5fe583cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:a5b8907a5fe583cb"}
13:07:48 [13:07:48.974]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a5b8907a5fe583cb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a5b8907a5fe583cb"}
13:07:48 [13:07:48.974]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a5b8907a5fe583cb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a5b8907a5fe583cb"}
13:07:48 [13:07:48.975]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a5b8907a5fe583cb jobErr=test error: Error: db error
13:07:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:a5b8907a5fe583cb"}
13:07:48 [13:07:48.976]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.977]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:30a7f744e21f8775 not found
{"provingJobId":"1:BASE_PARITY:30a7f744e21f8775"}
13:07:48 [13:07:48.979]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.979]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:773ea384d10a5ebe err=test error
{"provingJoId":"1:BASE_PARITY:773ea384d10a5ebe"}
13:07:48 [13:07:48.981]
INFO:
prover-client:proving-broker Proving Broker started
13:07:48 [13:07:48.991]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b79be76c092a6ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b79be76c092a6ec"}
13:07:48 [13:07:48.992]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8b79be76c092a6ec type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8b79be76c092a6ec"}
13:07:48 [13:07:48.992]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8b79be76c092a6ec type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8b79be76c092a6ec"}
13:07:49 [13:07:49.003]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:2bc8d7f4cb7da92f epochNumber=2
{"provingJobId":"2:BASE_PARITY:2bc8d7f4cb7da92f"}
13:07:49 [13:07:49.004]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:2bc8d7f4cb7da92f type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:2bc8d7f4cb7da92f"}
13:07:49 [13:07:49.004]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2bc8d7f4cb7da92f type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:2bc8d7f4cb7da92f"}
13:07:49 [13:07:49.066]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:63071267e480a7ac epochNumber=3
{"provingJobId":"3:BASE_PARITY:63071267e480a7ac"}
13:07:49 [13:07:49.083]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:49 [13:07:49.128]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:c782f7b5e0ef7c44 epochNumber=4
{"provingJobId":"4:BASE_PARITY:c782f7b5e0ef7c44"}
13:07:49 [13:07:49.134]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:49 [13:07:49.191]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:5965040b0d6124ab epochNumber=5
{"provingJobId":"5:BASE_PARITY:5965040b0d6124ab"}
13:07:49 [13:07:49.210]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:49 [13:07:49.252]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ad3773825a24adf7 epochNumber=42
{"provingJobId":"1:BASE_PARITY:ad3773825a24adf7"}
13:07:49 [13:07:49.254]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testKElrOh/42 with map size 1073741824
13:07:49 [13:07:49.261]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.290]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:db7bfb4d5169bcf2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:db7bfb4d5169bcf2"}
13:07:49 [13:07:49.294]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.295]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1bcd090d5396336c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1bcd090d5396336c"}
13:07:49 [13:07:49.296]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJqIBkl/1 with map size 1073741824
13:07:49 [13:07:49.299]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.318]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51d75afec15c21ce epochNumber=1
{"provingJobId":"1:BASE_PARITY:51d75afec15c21ce"}
13:07:49 [13:07:49.327]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.327]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52b19dc3de22a183 epochNumber=1
{"provingJobId":"1:BASE_PARITY:52b19dc3de22a183"}
13:07:49 [13:07:49.328]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0Mt8XT/1 with map size 1073741824
13:07:49 [13:07:49.330]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.350]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:52b19dc3de22a183 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:52b19dc3de22a183"}
13:07:49 [13:07:49.357]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.359]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:69960895b59ba44d epochNumber=1
{"provingJobId":"1:BASE_PARITY:69960895b59ba44d"}
13:07:49 [13:07:49.360]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhQpjoy/1 with map size 1073741824
13:07:49 [13:07:49.365]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.383]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:69960895b59ba44d epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:69960895b59ba44d"}
13:07:49 [13:07:49.384]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:69960895b59ba44d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:69960895b59ba44d"}
13:07:49 [13:07:49.391]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:69960895b59ba44d epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:69960895b59ba44d"}
13:07:49 [13:07:49.400]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.402]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b70974d7b3185f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b70974d7b3185f9"}
13:07:49 [13:07:49.403]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlpKDie/1 with map size 1073741824
13:07:49 [13:07:49.406]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.434]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.437]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.438]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45ab180066124b43 epochNumber=1
{"provingJobId":"1:BASE_PARITY:45ab180066124b43"}
13:07:49 [13:07:49.439]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLhDslZ/1 with map size 1073741824
13:07:49 [13:07:49.441]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.464]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:45ab180066124b43
{"provingJobId":"1:BASE_PARITY:45ab180066124b43"}
13:07:49 [13:07:49.465]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:45ab180066124b43 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:45ab180066124b43"}
13:07:49 [13:07:49.465]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:45ab180066124b43 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:45ab180066124b43"}
13:07:49 [13:07:49.474]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.475]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66d8f78c2c289ac2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66d8f78c2c289ac2"}
13:07:49 [13:07:49.476]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLOFJKY/1 with map size 1073741824
13:07:49 [13:07:49.477]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.497]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:66d8f78c2c289ac2
{"provingJobId":"1:BASE_PARITY:66d8f78c2c289ac2"}
13:07:49 [13:07:49.498]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:66d8f78c2c289ac2 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:66d8f78c2c289ac2"}
13:07:49 [13:07:49.506]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.507]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8936d7777833250b epochNumber=1
{"provingJobId":"1:BASE_PARITY:8936d7777833250b"}
13:07:49 [13:07:49.507]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxaIK0l/1 with map size 1073741824
13:07:49 [13:07:49.509]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.525]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8936d7777833250b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8936d7777833250b"}
13:07:49 [13:07:49.525]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8936d7777833250b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8936d7777833250b"}
13:07:49 [13:07:49.534]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:73bf80388a79ce00 epochNumber=1
{"provingJobId":"1:BASE_PARITY:73bf80388a79ce00"}
13:07:49 [13:07:49.536]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbHDs6m/1 with map size 1073741824
13:07:49 [13:07:49.538]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.562]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:73bf80388a79ce00 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:73bf80388a79ce00"}
13:07:49 [13:07:49.563]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:73bf80388a79ce00 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:73bf80388a79ce00"}
13:07:49 [13:07:49.573]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.573]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84fd10a06812ce62 epochNumber=0
{"provingJobId":"1:BASE_PARITY:84fd10a06812ce62"}
13:07:49 [13:07:49.574]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testHv282b/0 with map size 1073741824
13:07:49 [13:07:49.576]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.592]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:84fd10a06812ce62 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:84fd10a06812ce62"}
13:07:49 [13:07:49.593]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9fcc0f9c6cd55dd epochNumber=0
{"provingJobId":"1:BASE_PARITY:a9fcc0f9c6cd55dd"}
13:07:49 [13:07:49.594]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:84fd10a06812ce62 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:84fd10a06812ce62"}
13:07:49 [13:07:49.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7071b7d3f24a835f epochNumber=0
{"provingJobId":"1:BASE_PARITY:7071b7d3f24a835f"}
13:07:49 [13:07:49.611]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.614]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.615]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:706084d3e010e56e epochNumber=2
{"provingJobId":"1:BASE_PARITY:706084d3e010e56e"}
13:07:49 [13:07:49.616]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testu9XxPH/2 with map size 1073741824
13:07:49 [13:07:49.618]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.634]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e33fc8b497224e8a epochNumber=1
{"provingJobId":"1:BASE_PARITY:e33fc8b497224e8a"}
13:07:49 [13:07:49.635]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testu9XxPH/1 with map size 1073741824
13:07:49 [13:07:49.637]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.669]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.670]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6993e7f4e67afc59 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6993e7f4e67afc59"}
13:07:49 [13:07:49.670]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfivkgT/1 with map size 1073741824
13:07:49 [13:07:49.673]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.694]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.695]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7021d8ed38098ad0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7021d8ed38098ad0"}
13:07:49 [13:07:49.695]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8lToy2/1 with map size 1073741824
13:07:49 [13:07:49.697]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.715]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7078154a1e7b4c99 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7078154a1e7b4c99"}
13:07:49 [13:07:49.721]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3889c21eb1b8e8e2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3889c21eb1b8e8e2"}
13:07:49 [13:07:49.722]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8lToy2/2 with map size 1073741824
13:07:49 [13:07:49.724]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.745]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f870cc3edf630cb1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f870cc3edf630cb1"}
13:07:49 [13:07:49.759]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.760]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10e431531d88ab51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:10e431531d88ab51"}
13:07:49 [13:07:49.761]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZdy91R/1 with map size 1073741824
13:07:49 [13:07:49.763]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.780]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ccc9e6ab9e68f5d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ccc9e6ab9e68f5d0"}
13:07:49 [13:07:49.787]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20e32be2319d635a epochNumber=2
{"provingJobId":"1:BASE_PARITY:20e32be2319d635a"}
13:07:49 [13:07:49.788]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZdy91R/2 with map size 1073741824
13:07:49 [13:07:49.790]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ddcf62ca5d8bbf6f epochNumber=1
{"provingJobId":"1:BASE_PARITY:ddcf62ca5d8bbf6f"}
13:07:49 [13:07:49.815]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa8cd7e2c7e6572f epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa8cd7e2c7e6572f"}
13:07:49 [13:07:49.816]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvNFad9/1 with map size 1073741824
13:07:49 [13:07:49.819]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.834]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34cee06d7ba9204c epochNumber=1
{"provingJobId":"1:BASE_PARITY:34cee06d7ba9204c"}
13:07:49 [13:07:49.840]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ad8426905e6b328 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9ad8426905e6b328"}
13:07:49 [13:07:49.841]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvNFad9/2 with map size 1073741824
13:07:49 [13:07:49.845]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f55b3d19befba038 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f55b3d19befba038"}
13:07:49 [13:07:49.871]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.872]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1386023a7b78d08 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c1386023a7b78d08"}
13:07:49 [13:07:49.873]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPVxC1R/1 with map size 1073741824
13:07:49 [13:07:49.875]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75473209e6a5b0ab epochNumber=1
{"provingJobId":"1:BASE_PARITY:75473209e6a5b0ab"}
13:07:49 [13:07:49.898]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c1386023a7b78d08 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c1386023a7b78d08"}
13:07:49 [13:07:49.907]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.908]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c13293ca032ba691 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c13293ca032ba691"}
13:07:49 [13:07:49.909]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcD8eI6/1 with map size 1073741824
13:07:49 [13:07:49.910]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.927]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7cc411998f677af epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7cc411998f677af"}
13:07:49 [13:07:49.933]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c13293ca032ba691 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:c13293ca032ba691"}
13:07:49 [13:07:49.945]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3f02718e42d3916 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3f02718e42d3916"}
13:07:49 [13:07:49.946]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testn4Qztm/1 with map size 1073741824
13:07:49 [13:07:49.952]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:49 [13:07:49.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eae6513ce89a0946 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eae6513ce89a0946"}
13:07:49 [13:07:49.980]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a3f02718e42d3916 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:a3f02718e42d3916"}
13:07:49 [13:07:49.985]
INFO:
prover-client:proving-broker Proving Broker started
13:07:49 [13:07:49.986]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3acfe7d837cec25f epochNumber=1
{"provingJobId":"1:BASE_PARITY:3acfe7d837cec25f"}
13:07:49 [13:07:49.986]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTcW1gL/1 with map size 1073741824
13:07:49 [13:07:49.989]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:50 [13:07:50.008]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3acfe7d837cec25f
{"provingJobId":"1:BASE_PARITY:3acfe7d837cec25f"}
13:07:50 [13:07:50.009]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3acfe7d837cec25f type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3acfe7d837cec25f"}
13:07:50 [13:07:50.014]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f26e9595b5267345 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f26e9595b5267345"}
13:07:50 [13:07:50.021]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3acfe7d837cec25f has already been completed
{"provingJobId":"1:BASE_PARITY:3acfe7d837cec25f"}
13:07:50 [13:07:50.026]
INFO:
prover-client:proving-broker Proving Broker started
13:07:50 [13:07:50.027]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81376474877bf50d epochNumber=1
{"provingJobId":"1:BASE_PARITY:81376474877bf50d"}
13:07:50 [13:07:50.029]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSZ9qk3/1 with map size 1073741824
13:07:50 [13:07:50.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:50 [13:07:50.052]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:605c3594c4e042d9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:605c3594c4e042d9"}
13:07:50 [13:07:50.053]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSZ9qk3/2 with map size 1073741824
13:07:50 [13:07:50.059]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:51 [13:07:51.130]
INFO:
prover-client:proving-broker Proving Broker started
13:07:51 [13:07:51.134]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:81376474877bf50d settled=false
{"provingJobId":"1:BASE_PARITY:81376474877bf50d","status":"pending"}
13:07:51 [13:07:51.136]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:605c3594c4e042d9 settled=false
{"provingJobId":"1:BASE_PARITY:605c3594c4e042d9","status":"pending"}
13:07:51 [13:07:51.136]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:81376474877bf50d type=BASE_PARITY startedAt=1750684070078 older agent has taken job
{"provingJobId":"1:BASE_PARITY:81376474877bf50d"}
13:07:51 [13:07:51.137]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:81376474877bf50d type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:81376474877bf50d"}
13:07:51 [13:07:51.140]
INFO:
prover-client:proving-broker Proving Broker started
13:07:51 [13:07:51.140]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0d9d7180fb5c7ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0d9d7180fb5c7ea"}
13:07:51 [13:07:51.142]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testta8fMV/1 with map size 1073741824
13:07:51 [13:07:51.144]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:51 [13:07:51.166]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0531144fecd7ca8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f0531144fecd7ca8"}
13:07:51 [13:07:51.167]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testta8fMV/2 with map size 1073741824
13:07:51 [13:07:51.169]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:52 [13:07:52.185]
INFO:
prover-client:proving-broker Proving Broker started
13:07:52 [13:07:52.187]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e0d9d7180fb5c7ea settled=false
{"provingJobId":"1:BASE_PARITY:e0d9d7180fb5c7ea","status":"pending"}
13:07:52 [13:07:52.188]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f0531144fecd7ca8 settled=false
{"provingJobId":"1:BASE_PARITY:f0531144fecd7ca8","status":"pending"}
13:07:52 [13:07:52.189]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e0d9d7180fb5c7ea type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:e0d9d7180fb5c7ea"}
13:07:52 [13:07:52.193]
INFO:
prover-client:proving-broker Proving Broker started
13:07:52 [13:07:52.194]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a97b0311217bd839 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a97b0311217bd839"}
13:07:52 [13:07:52.194]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlwMRpk/1 with map size 1073741824
13:07:52 [13:07:52.196]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:52 [13:07:52.221]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98d13994e8babcfd epochNumber=2
{"provingJobId":"1:BASE_PARITY:98d13994e8babcfd"}
13:07:52 [13:07:52.223]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlwMRpk/2 with map size 1073741824
13:07:52 [13:07:52.227]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:53 [13:07:53.250]
INFO:
prover-client:proving-broker Proving Broker started
13:07:53 [13:07:53.252]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a97b0311217bd839 settled=false
{"provingJobId":"1:BASE_PARITY:a97b0311217bd839","status":"pending"}
13:07:53 [13:07:53.254]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:98d13994e8babcfd settled=false
{"provingJobId":"1:BASE_PARITY:98d13994e8babcfd","status":"pending"}
13:07:53 [13:07:53.256]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a97b0311217bd839 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a97b0311217bd839"}
13:07:53 [13:07:53.256]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a97b0311217bd839 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a97b0311217bd839"}
13:07:53 [13:07:53.268]
INFO:
prover-client:proving-broker Proving Broker started
13:07:53 [13:07:53.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:329bd3f45907c9e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:329bd3f45907c9e5"}
13:07:53 [13:07:53.270]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRrmsFG/1 with map size 1073741824
13:07:53 [13:07:53.272]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:53 [13:07:53.290]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2eeba3db150f2bd epochNumber=2
{"provingJobId":"1:BASE_PARITY:c2eeba3db150f2bd"}
13:07:53 [13:07:53.291]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testRrmsFG/2 with map size 1073741824
13:07:53 [13:07:53.293]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:53 [13:07:53.312]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:329bd3f45907c9e5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:329bd3f45907c9e5"}
13:07:53 [13:07:53.318]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c2eeba3db150f2bd type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c2eeba3db150f2bd"}
13:07:53 [13:07:53.326]
INFO:
prover-client:proving-broker Proving Broker started
13:07:53 [13:07:53.327]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a645713162c8887 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a645713162c8887"}
13:07:53 [13:07:53.328]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKPyOB8/1 with map size 1073741824
13:07:53 [13:07:53.329]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:53 [13:07:53.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b5a96b63025e37f epochNumber=2
{"provingJobId":"1:BASE_PARITY:6b5a96b63025e37f"}
13:07:53 [13:07:53.347]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKPyOB8/2 with map size 1073741824
13:07:53 [13:07:53.351]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:53 [13:07:53.369]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7a645713162c8887 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7a645713162c8887"}
13:07:53 [13:07:53.370]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7a645713162c8887 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7a645713162c8887"}
13:07:53 [13:07:53.376]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6b5a96b63025e37f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6b5a96b63025e37f"}
13:07:53 [13:07:53.388]
INFO:
prover-client:proving-broker Proving Broker started
13:07:53 [13:07:53.389]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:60313ea02cb2d0c5 err=test error
{"provingJoId":"1:BASE_PARITY:60313ea02cb2d0c5"}
13:07:53 [13:07:53.394]
INFO:
prover-client:proving-broker Proving Broker started
13:07:53 [13:07:53.395]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8489b3ecbac7b5ae not found
{"provingJobId":"1:BASE_PARITY:8489b3ecbac7b5ae"}
13:07:53 [13:07:53.399]
INFO:
prover-client:proving-broker Proving Broker started
13:07:53 [13:07:53.400]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7304e223124d6cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7304e223124d6cb"}
13:07:53 [13:07:53.401]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXqvQt8/1 with map size 1073741824
13:07:53 [13:07:53.403]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:53 [13:07:53.421]
INFO:
prover-client:proving-broker Proving Broker started
13:07:53 [13:07:53.422]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd5992e32eaef7f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd5992e32eaef7f5"}
13:07:53 [13:07:53.423]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZv2hMB/1 with map size 1073741824
13:07:53 [13:07:53.425]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:53 [13:07:53.547]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dd5992e32eaef7f5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:dd5992e32eaef7f5"}
13:07:53 [13:07:53.551]
INFO:
prover-client:proving-broker Proving Broker started
13:07:53 [13:07:53.551]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71eb122b8853e308 epochNumber=1
{"provingJobId":"1:BASE_PARITY:71eb122b8853e308"}
13:07:53 [13:07:53.553]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMT1B2h/1 with map size 1073741824
13:07:53 [13:07:53.555]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:53 [13:07:53.678]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:71eb122b8853e308 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:71eb122b8853e308"}
13:07:53 [13:07:53.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a08079eb164439d epochNumber=10
{"provingJobId":"1:BASE_PARITY:5a08079eb164439d"}
13:07:53 [13:07:53.684]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testMT1B2h/10 with map size 1073741824
13:07:53 [13:07:53.687]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:53 [13:07:53.730]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:53 [13:07:53.730]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:07:53 [13:07:53.841]
INFO:
prover-client:proving-broker Proving Broker started
13:07:53 [13:07:53.842]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45c86d5c43c5fedd epochNumber=1
{"provingJobId":"1:BASE_PARITY:45c86d5c43c5fedd"}
13:07:53 [13:07:53.847]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpBnzQ9/1 with map size 1073741824
13:07:53 [13:07:53.855]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.042]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:45c86d5c43c5fedd timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:45c86d5c43c5fedd"}
13:07:54 [13:07:54.082]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da69f403313554a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:da69f403313554a7"}
13:07:54 [13:07:54.083]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIwjG5p/1 with map size 1073741824
13:07:54 [13:07:54.085]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.102]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:da69f403313554a7 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:da69f403313554a7"}
13:07:54 [13:07:54.105]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.106]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb518de52df6b59b epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb518de52df6b59b"}
13:07:54 [13:07:54.106]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMn1xLW/1 with map size 1073741824
13:07:54 [13:07:54.108]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.124]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bb518de52df6b59b type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:bb518de52df6b59b"}
13:07:54 [13:07:54.125]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bb518de52df6b59b type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:bb518de52df6b59b"}
13:07:54 [13:07:54.134]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.135]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13580fba9ffddc71 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13580fba9ffddc71"}
13:07:54 [13:07:54.136]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNnpElS/1 with map size 1073741824
13:07:54 [13:07:54.138]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.158]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:13580fba9ffddc71 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:13580fba9ffddc71"}
13:07:54 [13:07:54.169]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.170]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6c4ced107151292 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6c4ced107151292"}
13:07:54 [13:07:54.170]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnTd64k/1 with map size 1073741824
13:07:54 [13:07:54.173]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.191]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a6c4ced107151292 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a6c4ced107151292"}
13:07:54 [13:07:54.192]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0c170f909764439 epochNumber=3
{"provingJobId":"1:BASE_PARITY:b0c170f909764439"}
13:07:54 [13:07:54.192]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testnTd64k/3 with map size 1073741824
13:07:54 [13:07:54.194]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.220]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:54 [13:07:54.221]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:07:54 [13:07:54.236]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a6c4ced107151292 err=test error
{"provingJoId":"1:BASE_PARITY:a6c4ced107151292"}
13:07:54 [13:07:54.241]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmf358v/1 with map size 1073741824
13:07:54 [13:07:54.249]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.276]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testmf358v/2 with map size 1073741824
13:07:54 [13:07:54.279]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.297]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.300]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dcfe9dc62c91d384 settled=false
{"provingJobId":"1:BASE_PARITY:dcfe9dc62c91d384","status":"pending"}
13:07:54 [13:07:54.302]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:94cfaaca0a9427da settled=false
{"provingJobId":"1:BASE_PARITY:94cfaaca0a9427da","status":"pending"}
13:07:54 [13:07:54.308]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testviT36J/1 with map size 1073741824
13:07:54 [13:07:54.310]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.326]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testviT36J/2 with map size 1073741824
13:07:54 [13:07:54.329]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.364]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.367]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:afac6ae0ac08f753 settled=true
{"provingJobId":"1:BASE_PARITY:afac6ae0ac08f753","status":"fulfilled"}
13:07:54 [13:07:54.368]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:eb5826aead24347c settled=true
{"provingJobId":"2:BASE_PARITY:eb5826aead24347c","status":"fulfilled"}
13:07:54 [13:07:54.373]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5oSyIn/1 with map size 1073741824
13:07:54 [13:07:54.374]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.399]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5oSyIn/2 with map size 1073741824
13:07:54 [13:07:54.401]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.417]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.419]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:460631e6765e7f2d settled=true
{"provingJobId":"1:BASE_PARITY:460631e6765e7f2d","status":"fulfilled"}
13:07:54 [13:07:54.421]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ca45471b409ebde6 settled=false
{"provingJobId":"1:BASE_PARITY:ca45471b409ebde6","status":"pending"}
13:07:54 [13:07:54.428]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.429]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:963e5712594f7ad9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:963e5712594f7ad9"}
13:07:54 [13:07:54.429]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLGNCst/1 with map size 1073741824
13:07:54 [13:07:54.431]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.458]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b4327ef3133a0828 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b4327ef3133a0828"}
13:07:54 [13:07:54.460]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:b4327ef3133a0828: Error: db error: Error: db error
13:07:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:b4327ef3133a0828"}
13:07:54 [13:07:54.464]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.465]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e5493d06c4637d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e5493d06c4637d9"}
13:07:54 [13:07:54.466]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiBDVrs/1 with map size 1073741824
13:07:54 [13:07:54.468]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.485]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0e5493d06c4637d9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0e5493d06c4637d9"}
13:07:54 [13:07:54.486]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0e5493d06c4637d9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0e5493d06c4637d9"}
13:07:54 [13:07:54.497]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.498]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c388fdd74836b031 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c388fdd74836b031"}
13:07:54 [13:07:54.499]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhZMH8f/1 with map size 1073741824
13:07:54 [13:07:54.501]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.519]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c388fdd74836b031 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c388fdd74836b031"}
13:07:54 [13:07:54.520]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c388fdd74836b031 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c388fdd74836b031"}
13:07:54 [13:07:54.521]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:c388fdd74836b031: Error: db error
13:07:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:c388fdd74836b031"}
13:07:54 [13:07:54.525]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.526]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c752fe048342541 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c752fe048342541"}
13:07:54 [13:07:54.526]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5ffhJI/1 with map size 1073741824
13:07:54 [13:07:54.528]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.546]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8c752fe048342541 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8c752fe048342541"}
13:07:54 [13:07:54.546]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8c752fe048342541 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8c752fe048342541"}
13:07:54 [13:07:54.562]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.563]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b20a45daa01d63b epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b20a45daa01d63b"}
13:07:54 [13:07:54.564]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPviKg7/1 with map size 1073741824
13:07:54 [13:07:54.568]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.586]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b20a45daa01d63b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7b20a45daa01d63b"}
13:07:54 [13:07:54.587]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7b20a45daa01d63b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7b20a45daa01d63b"}
13:07:54 [13:07:54.588]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:7b20a45daa01d63b jobErr=test error: Error: db error
13:07:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:7b20a45daa01d63b"}
13:07:54 [13:07:54.593]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.594]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b40b23149f1fd878 not found
{"provingJobId":"1:BASE_PARITY:b40b23149f1fd878"}
13:07:54 [13:07:54.599]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.600]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:277281f6b2e3ce50 err=test error
{"provingJoId":"1:BASE_PARITY:277281f6b2e3ce50"}
13:07:54 [13:07:54.604]
INFO:
prover-client:proving-broker Proving Broker started
13:07:54 [13:07:54.616]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b4f861c44b79637 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5b4f861c44b79637"}
13:07:54 [13:07:54.617]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4JyXiM/1 with map size 1073741824
13:07:54 [13:07:54.621]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.637]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5b4f861c44b79637 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5b4f861c44b79637"}
13:07:54 [13:07:54.638]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5b4f861c44b79637 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5b4f861c44b79637"}
13:07:54 [13:07:54.657]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:6a0e37bef90e50ea epochNumber=2
{"provingJobId":"2:BASE_PARITY:6a0e37bef90e50ea"}
13:07:54 [13:07:54.659]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4JyXiM/2 with map size 1073741824
13:07:54 [13:07:54.672]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.695]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:6a0e37bef90e50ea type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:6a0e37bef90e50ea"}
13:07:54 [13:07:54.696]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:6a0e37bef90e50ea type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:6a0e37bef90e50ea"}
13:07:54 [13:07:54.767]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:e559a5a4b7877d89 epochNumber=3
{"provingJobId":"3:BASE_PARITY:e559a5a4b7877d89"}
13:07:54 [13:07:54.768]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test4JyXiM/3 with map size 1073741824
13:07:54 [13:07:54.776]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.809]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:54 [13:07:54.809]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
13:07:54 [13:07:54.858]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:0837f18bf81db5b2 epochNumber=4
{"provingJobId":"4:BASE_PARITY:0837f18bf81db5b2"}
13:07:54 [13:07:54.859]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-test4JyXiM/4 with map size 1073741824
13:07:54 [13:07:54.866]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.890]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:54 [13:07:54.890]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
13:07:54 [13:07:54.950]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:f115445ed46977aa epochNumber=5
{"provingJobId":"5:BASE_PARITY:f115445ed46977aa"}
13:07:54 [13:07:54.952]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-test4JyXiM/5 with map size 1073741824
13:07:54 [13:07:54.957]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
13:07:54 [13:07:54.995]
INFO:
prover-client:proving-broker Cleaned up jobs=1
13:07:54 [13:07:54.996]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
13:07:55
PASS src/proving_broker/proving_broker.test.ts (
13.109 s)
13:07:55 ProvingBroker
13:07:55 Producer API
13:07:55
✓ refuses stale jobs (42 ms)
13:07:55
✓ enqueues jobs (4 ms)
13:07:55
✓ ignores duplicate jobs (4 ms)
13:07:55
✓ reports correct status when enqueuing repeat jobs (4 ms)
13:07:55
✓ throws an error in case of duplicate job IDs (25 ms)
13:07:55
✓ returns not-found status for non-existing jobs (2 ms)
13:07:55
✓ cancels jobs in queue (3 ms)
13:07:55
✓ cancels jobs in-progress (3 ms)
13:07:55
✓ returns job result if successful (3 ms)
13:07:55
✓ returns job error if failed (3 ms)
13:07:55
✓ correctly returns job status for concurrent writes (3 ms)
13:07:55
✓ refuses stale jobs (49 ms)
13:07:55
✓ enqueues jobs (32 ms)
13:07:55
✓ ignores duplicate jobs (30 ms)
13:07:55
✓ reports correct status when enqueuing repeat jobs (38 ms)
13:07:55
✓ throws an error in case of duplicate job IDs (37 ms)
13:07:55
✓ returns not-found status for non-existing jobs (4 ms)
13:07:55
✓ cancels jobs in queue (37 ms)
13:07:55
✓ cancels jobs in-progress (31 ms)
13:07:55
✓ returns job result if successful (27 ms)
13:07:55
✓ returns job error if failed (39 ms)
13:07:55
✓ correctly returns job status for concurrent writes (37 ms)
13:07:55 Consumer API
13:07:55
✓ returns undefined if no jobs are available (1 ms)
13:07:55
✓ returns jobs in priority order (2 ms)
13:07:55
✓ returns undefined if no jobs are available for the given allowList (2 ms)
13:07:55
✓ returns a job if it is in the allowList (2 ms)
13:07:55
✓ returns the most important job if it is in the allowList (2 ms)
13:07:55
✓ returns any job if filter is empty (2 ms)
13:07:55
✓ returns a new job when reporting job success (2 ms)
13:07:55
✓ returns a new job when reporting permanent error (2 ms)
13:07:55
✓ returns a new job when reporting retry-able error (2 ms)
13:07:55
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
13:07:55
✓ returns a new job if job is already in progress elsewhere (1057 ms)
13:07:55
✓ avoids sending the same job to a new agent after a restart (1005 ms)
13:07:55
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
13:07:55
✓ tracks job result if in progress (3 ms)
13:07:55
✓ tracks job result even if job is in queue (3 ms)
13:07:55
✓ ignores reported job error if unknown job (1 ms)
13:07:55
✓ ignores job result if unknown job (1 ms)
13:07:55
✓ returns undefined if no jobs are available (3 ms)
13:07:55
✓ returns jobs in priority order (54 ms)
13:07:55
✓ returns undefined if no jobs are available for the given allowList (24 ms)
13:07:55
✓ returns a job if it is in the allowList (63 ms)
13:07:55
✓ returns the most important job if it is in the allowList (56 ms)
13:07:55
✓ returns any job if filter is empty (56 ms)
13:07:55
✓ returns a new job when reporting job success (35 ms)
13:07:55
✓ returns a new job when reporting permanent error (34 ms)
13:07:55
✓ returns a new job when reporting retry-able error (42 ms)
13:07:55
✓ returns a new job when reporting progress if current one is cancelled (41 ms)
13:07:55
✓ returns a new job if job is already in progress elsewhere (1114 ms)
13:07:55
✓ avoids sending the same job to a new agent after a restart (1053 ms)
13:07:55
✓ avoids sending a completed job to a new agent after a restart (1075 ms)
13:07:55
✓ tracks job result if in progress (57 ms)
13:07:55
✓ tracks job result even if job is in queue (60 ms)
13:07:55
✓ ignores reported job error if unknown job (6 ms)
13:07:55
✓ ignores job result if unknown job (5 ms)
13:07:55 Timeouts
13:07:55
✓ tracks in progress jobs (2 ms)
13:07:55
✓ re-enqueues jobs that time out (111 ms)
13:07:55
✓ cancel stale jobs that time out (233 ms)
13:07:55
✓ keeps the jobs in progress while it is alive (205 ms)
13:07:55
✓ tracks in progress jobs (22 ms)
13:07:55
✓ re-enqueues jobs that time out (129 ms)
13:07:55
✓ cancel stale jobs that time out (288 ms)
13:07:55
✓ keeps the jobs in progress while it is alive (242 ms)
13:07:55 Retries
13:07:55
✓ retries jobs (3 ms)
13:07:55
✓ retries up to a maximum number of times (3 ms)
13:07:55
✓ passing retry=false does not retry (2 ms)
13:07:55
✓ does not retry if job is stale (28 ms)
13:07:55
✓ retries jobs (23 ms)
13:07:55
✓ retries up to a maximum number of times (29 ms)
13:07:55
✓ passing retry=false does not retry (33 ms)
13:07:55
✓ does not retry if job is stale (71 ms)
13:07:55 Database management
13:07:55
✓ re-enqueues proof requests on start (3 ms)
13:07:55
✓ restores proof results on start (2 ms)
13:07:55
✓ only re-enqueues unfinished jobs (10 ms)
13:07:55
✓ saves job when enqueued (5 ms)
13:07:55
✓ does not retain job if database fails to save (4 ms)
13:07:55
✓ saves job result (4 ms)
13:07:55
✓ saves result even if database fails to save (2 ms)
13:07:55
✓ saves job error (2 ms)
13:07:55
✓ saves job error even if database fails to save (2 ms)
13:07:55
✓ does not save job result if job is unknown (2 ms)
13:07:55
✓ does not save job error if job is unknown (2 ms)
13:07:55
✓ cleans up old jobs periodically (264 ms)
13:07:55
✓ re-enqueues proof requests on start (67 ms)
13:07:55
✓ restores proof results on start (64 ms)
13:07:55
✓ only re-enqueues unfinished jobs (55 ms)
13:07:55
✓ saves job when enqueued (28 ms)
13:07:55
✓ does not retain job if database fails to save (7 ms)
13:07:55
✓ saves job result (32 ms)
13:07:55
✓ saves result even if database fails to save (27 ms)
13:07:55
✓ saves job error (33 ms)
13:07:55
✓ saves job error even if database fails to save (33 ms)
13:07:55
✓ does not save job result if job is unknown (6 ms)
13:07:55
✓ does not save job error if job is unknown (5 ms)
13:07:55
✓ cleans up old jobs periodically (429 ms)
13:07:55
13:07:55
Test Suites: 1 passed, 1 total
13:07:55
Tests: 96 passed, 96 total
13:07:55
Snapshots: 0 total
13:07:55
Time: 13.226 s
13:07:55
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
13:07:55
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?