Parent Log:
http://ci.aztec-labs.com/8ed5c55c2bfee19e
Command: 9b90ba1cd25aa9c9 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/4ae29efe4b68ec58d68cf3708b9a36b94dad8b2b
Env: REF_NAME=gh-readonly-queue/next/pr-15129-5c77169d7f01a9e15bc19b05fa231fd96111d7dd CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Wed Jun 18 15:18:44 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15129_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
15:18:48 [15:18:48.624]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.631]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82d53adc5d48e0ee epochNumber=42
{"provingJobId":"1:BASE_PARITY:82d53adc5d48e0ee"}
15:18:48 [15:18:48.635]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:dad53e8a6fa629ab epochNumber=1
{"provingJobId":"1:BASE_PARITY:dad53e8a6fa629ab"}
15:18:48 [15:18:48.673]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.674]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:627f2bc6bf97c62a epochNumber=1
{"provingJobId":"1:BASE_PARITY:627f2bc6bf97c62a"}
15:18:48 [15:18:48.676]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6307cf2d9d4e3757 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6307cf2d9d4e3757"}
15:18:48 [15:18:48.678]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.678]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4e92826c3c3ba26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4e92826c3c3ba26"}
15:18:48 [15:18:48.679]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d4e92826c3c3ba26 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d4e92826c3c3ba26"}
15:18:48 [15:18:48.682]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26e1c807481ca969 epochNumber=1
{"provingJobId":"1:BASE_PARITY:26e1c807481ca969"}
15:18:48 [15:18:48.684]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:26e1c807481ca969 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:26e1c807481ca969"}
15:18:48 [15:18:48.684]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:26e1c807481ca969 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:26e1c807481ca969"}
15:18:48 [15:18:48.685]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:26e1c807481ca969 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:26e1c807481ca969"}
15:18:48 [15:18:48.686]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.687]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4671b1d216d6ca7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e4671b1d216d6ca7"}
15:18:48 [15:18:48.723]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.725]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.725]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9412ff666d1a5aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9412ff666d1a5aa"}
15:18:48 [15:18:48.726]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e9412ff666d1a5aa
{"provingJobId":"1:BASE_PARITY:e9412ff666d1a5aa"}
15:18:48 [15:18:48.727]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e9412ff666d1a5aa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e9412ff666d1a5aa"}
15:18:48 [15:18:48.727]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e9412ff666d1a5aa type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:e9412ff666d1a5aa"}
15:18:48 [15:18:48.729]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.729]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:225211ee9eb635b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:225211ee9eb635b3"}
15:18:48 [15:18:48.730]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:225211ee9eb635b3
{"provingJobId":"1:BASE_PARITY:225211ee9eb635b3"}
15:18:48 [15:18:48.730]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:225211ee9eb635b3 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:225211ee9eb635b3"}
15:18:48 [15:18:48.732]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.732]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bac7c50e9ceceb0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:bac7c50e9ceceb0f"}
15:18:48 [15:18:48.733]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bac7c50e9ceceb0f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bac7c50e9ceceb0f"}
15:18:48 [15:18:48.733]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bac7c50e9ceceb0f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bac7c50e9ceceb0f"}
15:18:48 [15:18:48.742]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0e2e4237fd93cc0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c0e2e4237fd93cc0"}
15:18:48 [15:18:48.743]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c0e2e4237fd93cc0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c0e2e4237fd93cc0"}
15:18:48 [15:18:48.743]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c0e2e4237fd93cc0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c0e2e4237fd93cc0"}
15:18:48 [15:18:48.744]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.745]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f201f375a1124ef epochNumber=0
{"provingJobId":"1:BASE_PARITY:5f201f375a1124ef"}
15:18:48 [15:18:48.745]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5f201f375a1124ef epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5f201f375a1124ef"}
15:18:48 [15:18:48.745]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41dc0c2cbd600fad epochNumber=0
{"provingJobId":"1:BASE_PARITY:41dc0c2cbd600fad"}
15:18:48 [15:18:48.746]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5f201f375a1124ef epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5f201f375a1124ef"}
15:18:48 [15:18:48.746]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eea28a75b98b9413 epochNumber=0
{"provingJobId":"1:BASE_PARITY:eea28a75b98b9413"}
15:18:48 [15:18:48.747]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.748]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.749]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:008766cc1e8e1a26 epochNumber=2
{"provingJobId":"1:BASE_PARITY:008766cc1e8e1a26"}
15:18:48 [15:18:48.749]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6644da221ddd00e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6644da221ddd00e2"}
15:18:48 [15:18:48.750]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.751]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93fc889de57c89bd epochNumber=1
{"provingJobId":"1:BASE_PARITY:93fc889de57c89bd"}
15:18:48 [15:18:48.752]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.752]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:073db46e3431e1aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:073db46e3431e1aa"}
15:18:48 [15:18:48.752]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be9063ad83a93a5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:be9063ad83a93a5e"}
15:18:48 [15:18:48.753]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:043f471f59d60293 epochNumber=2
{"provingJobId":"1:BASE_PARITY:043f471f59d60293"}
15:18:48 [15:18:48.753]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87afd38e7d0cc3b7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87afd38e7d0cc3b7"}
15:18:48 [15:18:48.754]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.755]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76a2d1c83148688d epochNumber=1
{"provingJobId":"1:BASE_PARITY:76a2d1c83148688d"}
15:18:48 [15:18:48.755]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5886e24c08880481 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5886e24c08880481"}
15:18:48 [15:18:48.755]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d03428eb30ca552 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1d03428eb30ca552"}
15:18:48 [15:18:48.755]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:768e1467aa25751b epochNumber=1
{"provingJobId":"1:BASE_PARITY:768e1467aa25751b"}
15:18:48 [15:18:48.756]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.757]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95716ecc75d576e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:95716ecc75d576e2"}
15:18:48 [15:18:48.757]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45e8523c49eb4610 epochNumber=1
{"provingJobId":"1:BASE_PARITY:45e8523c49eb4610"}
15:18:48 [15:18:48.758]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8739dfe7a2e8e46f epochNumber=2
{"provingJobId":"1:BASE_PARITY:8739dfe7a2e8e46f"}
15:18:48 [15:18:48.758]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ed6394425a7c8c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ed6394425a7c8c0"}
15:18:48 [15:18:48.759]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.759]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7052f1b4e3b01003 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7052f1b4e3b01003"}
15:18:48 [15:18:48.760]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d8964ce400c19a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d8964ce400c19a0"}
15:18:48 [15:18:48.760]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7052f1b4e3b01003 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7052f1b4e3b01003"}
15:18:48 [15:18:48.761]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d93ce4c82f51086a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d93ce4c82f51086a"}
15:18:48 [15:18:48.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:577e05868dd7f5eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:577e05868dd7f5eb"}
15:18:48 [15:18:48.763]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d93ce4c82f51086a type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:d93ce4c82f51086a"}
15:18:48 [15:18:48.764]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.764]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4dd55ebacb4d9f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4dd55ebacb4d9f9"}
15:18:48 [15:18:48.765]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:932baf2d7ce104ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:932baf2d7ce104ed"}
15:18:48 [15:18:48.765]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f4dd55ebacb4d9f9 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:f4dd55ebacb4d9f9"}
15:18:48 [15:18:48.766]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.766]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d9073f4d74e9787 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5d9073f4d74e9787"}
15:18:48 [15:18:48.767]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5d9073f4d74e9787
{"provingJobId":"1:BASE_PARITY:5d9073f4d74e9787"}
15:18:48 [15:18:48.767]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5d9073f4d74e9787 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5d9073f4d74e9787"}
15:18:48 [15:18:48.767]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae7572ace3b90f37 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae7572ace3b90f37"}
15:18:48 [15:18:48.768]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5d9073f4d74e9787 has already been completed
{"provingJobId":"1:BASE_PARITY:5d9073f4d74e9787"}
15:18:48 [15:18:48.769]
INFO:
prover-client:proving-broker Proving Broker started
15:18:48 [15:18:48.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12c14a486a407f34 epochNumber=1
{"provingJobId":"1:BASE_PARITY:12c14a486a407f34"}
15:18:48 [15:18:48.770]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc879412b4db4b5a epochNumber=2
{"provingJobId":"1:BASE_PARITY:cc879412b4db4b5a"}
15:18:49 [15:18:49.821]
INFO:
prover-client:proving-broker Proving Broker started
15:18:49 [15:18:49.822]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:12c14a486a407f34 settled=false
{"provingJobId":"1:BASE_PARITY:12c14a486a407f34","status":"pending"}
15:18:49 [15:18:49.823]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cc879412b4db4b5a settled=false
{"provingJobId":"1:BASE_PARITY:cc879412b4db4b5a","status":"pending"}
15:18:49 [15:18:49.823]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:12c14a486a407f34 type=BASE_PARITY startedAt=1750259928770 older agent has taken job
{"provingJobId":"1:BASE_PARITY:12c14a486a407f34"}
15:18:49 [15:18:49.824]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:12c14a486a407f34 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:12c14a486a407f34"}
15:18:49 [15:18:49.825]
INFO:
prover-client:proving-broker Proving Broker started
15:18:49 [15:18:49.826]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c04737f1cc4bfda4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c04737f1cc4bfda4"}
15:18:49 [15:18:49.826]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f122c14732e425d5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f122c14732e425d5"}
15:18:50 [15:18:50.827]
INFO:
prover-client:proving-broker Proving Broker started
15:18:50 [15:18:50.827]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c04737f1cc4bfda4 settled=false
{"provingJobId":"1:BASE_PARITY:c04737f1cc4bfda4","status":"pending"}
15:18:50 [15:18:50.828]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f122c14732e425d5 settled=false
{"provingJobId":"1:BASE_PARITY:f122c14732e425d5","status":"pending"}
15:18:50 [15:18:50.829]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c04737f1cc4bfda4 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:c04737f1cc4bfda4"}
15:18:50 [15:18:50.832]
INFO:
prover-client:proving-broker Proving Broker started
15:18:50 [15:18:50.833]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e0204571660e82a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e0204571660e82a"}
15:18:50 [15:18:50.833]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2f2ddb34448a886 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c2f2ddb34448a886"}
15:18:51 [15:18:51.836]
INFO:
prover-client:proving-broker Proving Broker started
15:18:51 [15:18:51.838]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6e0204571660e82a settled=false
{"provingJobId":"1:BASE_PARITY:6e0204571660e82a","status":"pending"}
15:18:51 [15:18:51.839]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c2f2ddb34448a886 settled=false
{"provingJobId":"1:BASE_PARITY:c2f2ddb34448a886","status":"pending"}
15:18:51 [15:18:51.839]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6e0204571660e82a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6e0204571660e82a"}
15:18:51 [15:18:51.840]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6e0204571660e82a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6e0204571660e82a"}
15:18:51 [15:18:51.841]
INFO:
prover-client:proving-broker Proving Broker started
15:18:51 [15:18:51.842]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a678eabea41c8747 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a678eabea41c8747"}
15:18:51 [15:18:51.842]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0dc2827a5432a460 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0dc2827a5432a460"}
15:18:51 [15:18:51.843]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a678eabea41c8747 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a678eabea41c8747"}
15:18:51 [15:18:51.844]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0dc2827a5432a460 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0dc2827a5432a460"}
15:18:51 [15:18:51.846]
INFO:
prover-client:proving-broker Proving Broker started
15:18:51 [15:18:51.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d0e782d50fbf004 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d0e782d50fbf004"}
15:18:51 [15:18:51.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f642b63db42d0f5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7f642b63db42d0f5"}
15:18:51 [15:18:51.847]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4d0e782d50fbf004 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4d0e782d50fbf004"}
15:18:51 [15:18:51.848]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4d0e782d50fbf004 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4d0e782d50fbf004"}
15:18:51 [15:18:51.848]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7f642b63db42d0f5 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7f642b63db42d0f5"}
15:18:51 [15:18:51.853]
INFO:
prover-client:proving-broker Proving Broker started
15:18:51 [15:18:51.854]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:47529e83552b4508 err=test error
{"provingJoId":"1:BASE_PARITY:47529e83552b4508"}
15:18:51 [15:18:51.855]
INFO:
prover-client:proving-broker Proving Broker started
15:18:51 [15:18:51.855]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:238fd52156f7881b not found
{"provingJobId":"1:BASE_PARITY:238fd52156f7881b"}
15:18:51 [15:18:51.857]
INFO:
prover-client:proving-broker Proving Broker started
15:18:51 [15:18:51.859]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0a1a5a130d38f4e epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0a1a5a130d38f4e"}
15:18:51 [15:18:51.861]
INFO:
prover-client:proving-broker Proving Broker started
15:18:51 [15:18:51.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f57c254e408585d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f57c254e408585d4"}
15:18:51 [15:18:51.964]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f57c254e408585d4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f57c254e408585d4"}
15:18:51 [15:18:51.970]
INFO:
prover-client:proving-broker Proving Broker started
15:18:51 [15:18:51.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58831438f01d55b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:58831438f01d55b5"}
15:18:52 [15:18:52.072]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:58831438f01d55b5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:58831438f01d55b5"}
15:18:52 [15:18:52.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc7b6fb19fae73a2 epochNumber=10
{"provingJobId":"1:BASE_PARITY:dc7b6fb19fae73a2"}
15:18:52 [15:18:52.099]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:52 [15:18:52.200]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.201]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ba2db8553b9c035 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ba2db8553b9c035"}
15:18:52 [15:18:52.353]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5ba2db8553b9c035 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5ba2db8553b9c035"}
15:18:52 [15:18:52.407]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.408]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51b4f7cb469c63b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:51b4f7cb469c63b8"}
15:18:52 [15:18:52.409]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:51b4f7cb469c63b8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:51b4f7cb469c63b8"}
15:18:52 [15:18:52.410]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.411]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de3af63e6d630e26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de3af63e6d630e26"}
15:18:52 [15:18:52.411]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:de3af63e6d630e26 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:de3af63e6d630e26"}
15:18:52 [15:18:52.412]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:de3af63e6d630e26 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:de3af63e6d630e26"}
15:18:52 [15:18:52.413]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.414]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12438bb10e912b51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:12438bb10e912b51"}
15:18:52 [15:18:52.414]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:12438bb10e912b51 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:12438bb10e912b51"}
15:18:52 [15:18:52.416]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.416]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:165ff487fbdd6095 epochNumber=1
{"provingJobId":"1:BASE_PARITY:165ff487fbdd6095"}
15:18:52 [15:18:52.417]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:165ff487fbdd6095 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:165ff487fbdd6095"}
15:18:52 [15:18:52.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a724468c823a86e2 epochNumber=3
{"provingJobId":"1:BASE_PARITY:a724468c823a86e2"}
15:18:52 [15:18:52.441]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:52 [15:18:52.443]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:165ff487fbdd6095 err=test error
{"provingJoId":"1:BASE_PARITY:165ff487fbdd6095"}
15:18:52 [15:18:52.445]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.445]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b02404ba32d1465a settled=false
{"provingJobId":"1:BASE_PARITY:b02404ba32d1465a","status":"pending"}
15:18:52 [15:18:52.445]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:32d504dc11889a1d settled=false
{"provingJobId":"1:BASE_PARITY:32d504dc11889a1d","status":"pending"}
15:18:52 [15:18:52.447]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.447]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:03438d81333de476 settled=true
{"provingJobId":"1:BASE_PARITY:03438d81333de476","status":"fulfilled"}
15:18:52 [15:18:52.448]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:bb982b39fd520ef0 settled=true
{"provingJobId":"2:BASE_PARITY:bb982b39fd520ef0","status":"fulfilled"}
15:18:52 [15:18:52.449]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.449]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ae64e4617b01ac0f settled=true
{"provingJobId":"1:BASE_PARITY:ae64e4617b01ac0f","status":"fulfilled"}
15:18:52 [15:18:52.450]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:168c42107cd5c5fc settled=false
{"provingJobId":"1:BASE_PARITY:168c42107cd5c5fc","status":"pending"}
15:18:52 [15:18:52.451]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fad1a20c91705e3b epochNumber=1
{"provingJobId":"1:BASE_PARITY:fad1a20c91705e3b"}
15:18:52 [15:18:52.454]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.454]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:efb3fc8f3a904ad1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:efb3fc8f3a904ad1"}
15:18:52 [15:18:52.455]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:efb3fc8f3a904ad1: Error: db error: Error: db error
15:18:52 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:efb3fc8f3a904ad1"}
15:18:52 [15:18:52.456]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.456]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80c859e4fd774e6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:80c859e4fd774e6a"}
15:18:52 [15:18:52.457]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:80c859e4fd774e6a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:80c859e4fd774e6a"}
15:18:52 [15:18:52.457]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:80c859e4fd774e6a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:80c859e4fd774e6a"}
15:18:52 [15:18:52.458]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.458]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1f737cb3d5c9eb5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1f737cb3d5c9eb5"}
15:18:52 [15:18:52.459]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e1f737cb3d5c9eb5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e1f737cb3d5c9eb5"}
15:18:52 [15:18:52.459]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e1f737cb3d5c9eb5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e1f737cb3d5c9eb5"}
15:18:52 [15:18:52.460]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:e1f737cb3d5c9eb5: Error: db error
15:18:52 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:e1f737cb3d5c9eb5"}
15:18:52 [15:18:52.461]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.461]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:119c0eb45300858b epochNumber=1
{"provingJobId":"1:BASE_PARITY:119c0eb45300858b"}
15:18:52 [15:18:52.462]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:119c0eb45300858b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:119c0eb45300858b"}
15:18:52 [15:18:52.462]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:119c0eb45300858b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:119c0eb45300858b"}
15:18:52 [15:18:52.463]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.463]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5a2e10ee32fb615 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5a2e10ee32fb615"}
15:18:52 [15:18:52.463]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b5a2e10ee32fb615 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b5a2e10ee32fb615"}
15:18:52 [15:18:52.464]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b5a2e10ee32fb615 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b5a2e10ee32fb615"}
15:18:52 [15:18:52.464]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:b5a2e10ee32fb615 jobErr=test error: Error: db error
15:18:52 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:b5a2e10ee32fb615"}
15:18:52 [15:18:52.465]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.466]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:06415f2f9bdf3439 not found
{"provingJobId":"1:BASE_PARITY:06415f2f9bdf3439"}
15:18:52 [15:18:52.467]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.467]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4b0e082c577c87ba err=test error
{"provingJoId":"1:BASE_PARITY:4b0e082c577c87ba"}
15:18:52 [15:18:52.468]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c32b9cd9fe5199ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:c32b9cd9fe5199ed"}
15:18:52 [15:18:52.479]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c32b9cd9fe5199ed type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c32b9cd9fe5199ed"}
15:18:52 [15:18:52.479]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c32b9cd9fe5199ed type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c32b9cd9fe5199ed"}
15:18:52 [15:18:52.491]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:0eac00b9b6f22a57 epochNumber=2
{"provingJobId":"2:BASE_PARITY:0eac00b9b6f22a57"}
15:18:52 [15:18:52.492]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:0eac00b9b6f22a57 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:0eac00b9b6f22a57"}
15:18:52 [15:18:52.492]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:0eac00b9b6f22a57 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:0eac00b9b6f22a57"}
15:18:52 [15:18:52.554]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:0a55d79c2b1c882b epochNumber=3
{"provingJobId":"3:BASE_PARITY:0a55d79c2b1c882b"}
15:18:52 [15:18:52.570]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:52 [15:18:52.617]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:e57221c9e7a295f4 epochNumber=4
{"provingJobId":"4:BASE_PARITY:e57221c9e7a295f4"}
15:18:52 [15:18:52.622]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:52 [15:18:52.681]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:fb461863f80d8b25 epochNumber=5
{"provingJobId":"5:BASE_PARITY:fb461863f80d8b25"}
15:18:52 [15:18:52.697]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:52 [15:18:52.736]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.737]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:726c3cf0b4c140b1 epochNumber=42
{"provingJobId":"1:BASE_PARITY:726c3cf0b4c140b1"}
15:18:52 [15:18:52.738]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testU0Bd9a/42 with map size 1073741824
15:18:52 [15:18:52.742]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:52 [15:18:52.771]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:9ca591bf090158e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ca591bf090158e1"}
15:18:52 [15:18:52.776]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cdfd1d7c5d50891 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0cdfd1d7c5d50891"}
15:18:52 [15:18:52.777]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testI6OLc5/1 with map size 1073741824
15:18:52 [15:18:52.780]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:52 [15:18:52.809]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c2587d8381d1b6f epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c2587d8381d1b6f"}
15:18:52 [15:18:52.819]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.820]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ac193745e1ee9a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ac193745e1ee9a4"}
15:18:52 [15:18:52.821]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testi8TK23/1 with map size 1073741824
15:18:52 [15:18:52.824]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:52 [15:18:52.841]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9ac193745e1ee9a4 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9ac193745e1ee9a4"}
15:18:52 [15:18:52.845]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c433c06bf186574f epochNumber=1
{"provingJobId":"1:BASE_PARITY:c433c06bf186574f"}
15:18:52 [15:18:52.847]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWDvq48/1 with map size 1073741824
15:18:52 [15:18:52.848]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:52 [15:18:52.867]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c433c06bf186574f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c433c06bf186574f"}
15:18:52 [15:18:52.867]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c433c06bf186574f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c433c06bf186574f"}
15:18:52 [15:18:52.873]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c433c06bf186574f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c433c06bf186574f"}
15:18:52 [15:18:52.878]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.879]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5688bee115c29b54 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5688bee115c29b54"}
15:18:52 [15:18:52.880]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjBHRt8/1 with map size 1073741824
15:18:52 [15:18:52.882]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:52 [15:18:52.915]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.918]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.919]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1f2f6c65871aec2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1f2f6c65871aec2"}
15:18:52 [15:18:52.919]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQhV1eK/1 with map size 1073741824
15:18:52 [15:18:52.921]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:52 [15:18:52.939]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e1f2f6c65871aec2
{"provingJobId":"1:BASE_PARITY:e1f2f6c65871aec2"}
15:18:52 [15:18:52.939]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e1f2f6c65871aec2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e1f2f6c65871aec2"}
15:18:52 [15:18:52.940]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e1f2f6c65871aec2 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:e1f2f6c65871aec2"}
15:18:52 [15:18:52.948]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.949]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77ecb1fc1e1531f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77ecb1fc1e1531f9"}
15:18:52 [15:18:52.950]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwoNz1A/1 with map size 1073741824
15:18:52 [15:18:52.952]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:52 [15:18:52.971]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:77ecb1fc1e1531f9
{"provingJobId":"1:BASE_PARITY:77ecb1fc1e1531f9"}
15:18:52 [15:18:52.971]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:77ecb1fc1e1531f9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:77ecb1fc1e1531f9"}
15:18:52 [15:18:52.982]
INFO:
prover-client:proving-broker Proving Broker started
15:18:52 [15:18:52.982]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:548defd002cb2e6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:548defd002cb2e6c"}
15:18:52 [15:18:52.983]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLris64/1 with map size 1073741824
15:18:52 [15:18:52.986]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.015]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:548defd002cb2e6c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:548defd002cb2e6c"}
15:18:53 [15:18:53.015]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:548defd002cb2e6c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:548defd002cb2e6c"}
15:18:53 [15:18:53.027]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.028]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8fa1e73c53cfb0aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:8fa1e73c53cfb0aa"}
15:18:53 [15:18:53.029]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testL6hYO5/1 with map size 1073741824
15:18:53 [15:18:53.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.053]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8fa1e73c53cfb0aa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8fa1e73c53cfb0aa"}
15:18:53 [15:18:53.053]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8fa1e73c53cfb0aa type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8fa1e73c53cfb0aa"}
15:18:53 [15:18:53.062]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d967f9a025b50e8c epochNumber=0
{"provingJobId":"1:BASE_PARITY:d967f9a025b50e8c"}
15:18:53 [15:18:53.063]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testiY5gOo/0 with map size 1073741824
15:18:53 [15:18:53.066]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.084]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d967f9a025b50e8c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d967f9a025b50e8c"}
15:18:53 [15:18:53.085]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99266b8df683146c epochNumber=0
{"provingJobId":"1:BASE_PARITY:99266b8df683146c"}
15:18:53 [15:18:53.086]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d967f9a025b50e8c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d967f9a025b50e8c"}
15:18:53 [15:18:53.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7cbb3757a0d4deba epochNumber=0
{"provingJobId":"1:BASE_PARITY:7cbb3757a0d4deba"}
15:18:53 [15:18:53.103]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.106]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.107]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc174b27b65d2752 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fc174b27b65d2752"}
15:18:53 [15:18:53.107]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwB6pdj/2 with map size 1073741824
15:18:53 [15:18:53.111]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.130]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0276607b80ecfd50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0276607b80ecfd50"}
15:18:53 [15:18:53.132]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwB6pdj/1 with map size 1073741824
15:18:53 [15:18:53.135]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.158]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.158]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8dc9432a77605efa epochNumber=1
{"provingJobId":"1:BASE_PARITY:8dc9432a77605efa"}
15:18:53 [15:18:53.159]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlL1P7f/1 with map size 1073741824
15:18:53 [15:18:53.161]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.183]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.183]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bce2a176e7b5a541 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bce2a176e7b5a541"}
15:18:53 [15:18:53.184]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDyBOmy/1 with map size 1073741824
15:18:53 [15:18:53.186]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.209]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8065c0507d4a32ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:8065c0507d4a32ba"}
15:18:53 [15:18:53.215]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4963848ab52543bb epochNumber=2
{"provingJobId":"1:BASE_PARITY:4963848ab52543bb"}
15:18:53 [15:18:53.216]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDyBOmy/2 with map size 1073741824
15:18:53 [15:18:53.218]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48b46217b2303c68 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48b46217b2303c68"}
15:18:53 [15:18:53.245]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.246]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2e08233c2b78bc7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2e08233c2b78bc7"}
15:18:53 [15:18:53.247]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqJOTNb/1 with map size 1073741824
15:18:53 [15:18:53.248]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76cb7f2f0ae0ee66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:76cb7f2f0ae0ee66"}
15:18:53 [15:18:53.270]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2bec11572a73bf2d epochNumber=2
{"provingJobId":"1:BASE_PARITY:2bec11572a73bf2d"}
15:18:53 [15:18:53.271]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqJOTNb/2 with map size 1073741824
15:18:53 [15:18:53.272]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.293]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55826cb1b0a0e641 epochNumber=1
{"provingJobId":"1:BASE_PARITY:55826cb1b0a0e641"}
15:18:53 [15:18:53.305]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.306]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:afc478db520307df epochNumber=1
{"provingJobId":"1:BASE_PARITY:afc478db520307df"}
15:18:53 [15:18:53.307]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1goya9/1 with map size 1073741824
15:18:53 [15:18:53.311]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.330]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ef8a990ec563776 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ef8a990ec563776"}
15:18:53 [15:18:53.336]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d05827815bfcead epochNumber=2
{"provingJobId":"1:BASE_PARITY:2d05827815bfcead"}
15:18:53 [15:18:53.337]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test1goya9/2 with map size 1073741824
15:18:53 [15:18:53.343]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.361]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5544a7b8e1d72ac3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5544a7b8e1d72ac3"}
15:18:53 [15:18:53.371]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.372]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a069305d9a93d3eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:a069305d9a93d3eb"}
15:18:53 [15:18:53.373]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test38ERnW/1 with map size 1073741824
15:18:53 [15:18:53.375]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.395]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a97134c257483138 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a97134c257483138"}
15:18:53 [15:18:53.402]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a069305d9a93d3eb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a069305d9a93d3eb"}
15:18:53 [15:18:53.415]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.416]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a19a81743c7959c epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a19a81743c7959c"}
15:18:53 [15:18:53.417]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCHxLo1/1 with map size 1073741824
15:18:53 [15:18:53.420]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d058ad5d9783f12 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d058ad5d9783f12"}
15:18:53 [15:18:53.444]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8a19a81743c7959c type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:8a19a81743c7959c"}
15:18:53 [15:18:53.452]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b862302cd4c9685 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b862302cd4c9685"}
15:18:53 [15:18:53.454]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgkeVTv/1 with map size 1073741824
15:18:53 [15:18:53.456]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.475]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e7dd2cb2f315afa epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e7dd2cb2f315afa"}
15:18:53 [15:18:53.482]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4b862302cd4c9685 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:4b862302cd4c9685"}
15:18:53 [15:18:53.489]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.490]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e172fcdfc63e30a epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e172fcdfc63e30a"}
15:18:53 [15:18:53.493]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZjRqwv/1 with map size 1073741824
15:18:53 [15:18:53.495]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.519]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4e172fcdfc63e30a
{"provingJobId":"1:BASE_PARITY:4e172fcdfc63e30a"}
15:18:53 [15:18:53.520]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4e172fcdfc63e30a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4e172fcdfc63e30a"}
15:18:53 [15:18:53.528]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c857a905b2e15c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c857a905b2e15c8"}
15:18:53 [15:18:53.534]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4e172fcdfc63e30a has already been completed
{"provingJobId":"1:BASE_PARITY:4e172fcdfc63e30a"}
15:18:53 [15:18:53.537]
INFO:
prover-client:proving-broker Proving Broker started
15:18:53 [15:18:53.538]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d9ce678c08bfc35b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d9ce678c08bfc35b"}
15:18:53 [15:18:53.541]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWhGFr1/1 with map size 1073741824
15:18:53 [15:18:53.544]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:53 [15:18:53.563]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab21d46f7bffd630 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ab21d46f7bffd630"}
15:18:53 [15:18:53.564]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWhGFr1/2 with map size 1073741824
15:18:53 [15:18:53.566]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:54 [15:18:54.633]
INFO:
prover-client:proving-broker Proving Broker started
15:18:54 [15:18:54.638]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d9ce678c08bfc35b settled=false
{"provingJobId":"1:BASE_PARITY:d9ce678c08bfc35b","status":"pending"}
15:18:54 [15:18:54.640]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ab21d46f7bffd630 settled=false
{"provingJobId":"1:BASE_PARITY:ab21d46f7bffd630","status":"pending"}
15:18:54 [15:18:54.641]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d9ce678c08bfc35b type=BASE_PARITY startedAt=1750259933582 older agent has taken job
{"provingJobId":"1:BASE_PARITY:d9ce678c08bfc35b"}
15:18:54 [15:18:54.642]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d9ce678c08bfc35b type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:d9ce678c08bfc35b"}
15:18:54 [15:18:54.645]
INFO:
prover-client:proving-broker Proving Broker started
15:18:54 [15:18:54.645]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df88a4649d319586 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df88a4649d319586"}
15:18:54 [15:18:54.648]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnVImar/1 with map size 1073741824
15:18:54 [15:18:54.653]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:54 [15:18:54.672]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20055f21b2552739 epochNumber=2
{"provingJobId":"1:BASE_PARITY:20055f21b2552739"}
15:18:54 [15:18:54.673]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnVImar/2 with map size 1073741824
15:18:54 [15:18:54.676]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:55 [15:18:55.699]
INFO:
prover-client:proving-broker Proving Broker started
15:18:55 [15:18:55.701]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:df88a4649d319586 settled=false
{"provingJobId":"1:BASE_PARITY:df88a4649d319586","status":"pending"}
15:18:55 [15:18:55.703]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:20055f21b2552739 settled=false
{"provingJobId":"1:BASE_PARITY:20055f21b2552739","status":"pending"}
15:18:55 [15:18:55.704]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:df88a4649d319586 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:df88a4649d319586"}
15:18:55 [15:18:55.708]
INFO:
prover-client:proving-broker Proving Broker started
15:18:55 [15:18:55.709]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a9475215f61d43d epochNumber=1
{"provingJobId":"1:BASE_PARITY:9a9475215f61d43d"}
15:18:55 [15:18:55.709]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCrct8G/1 with map size 1073741824
15:18:55 [15:18:55.711]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:55 [15:18:55.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f7dce2e311902a2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1f7dce2e311902a2"}
15:18:55 [15:18:55.730]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCrct8G/2 with map size 1073741824
15:18:55 [15:18:55.732]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:56 [15:18:56.749]
INFO:
prover-client:proving-broker Proving Broker started
15:18:56 [15:18:56.752]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9a9475215f61d43d settled=false
{"provingJobId":"1:BASE_PARITY:9a9475215f61d43d","status":"pending"}
15:18:56 [15:18:56.753]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1f7dce2e311902a2 settled=false
{"provingJobId":"1:BASE_PARITY:1f7dce2e311902a2","status":"pending"}
15:18:56 [15:18:56.754]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9a9475215f61d43d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9a9475215f61d43d"}
15:18:56 [15:18:56.755]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9a9475215f61d43d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9a9475215f61d43d"}
15:18:56 [15:18:56.767]
INFO:
prover-client:proving-broker Proving Broker started
15:18:56 [15:18:56.768]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4aff62892eaeeb45 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4aff62892eaeeb45"}
15:18:56 [15:18:56.769]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfpPd4F/1 with map size 1073741824
15:18:56 [15:18:56.771]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:56 [15:18:56.796]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4368fc50d870c6c2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4368fc50d870c6c2"}
15:18:56 [15:18:56.797]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfpPd4F/2 with map size 1073741824
15:18:56 [15:18:56.801]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:56 [15:18:56.821]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4aff62892eaeeb45 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4aff62892eaeeb45"}
15:18:56 [15:18:56.828]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4368fc50d870c6c2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4368fc50d870c6c2"}
15:18:56 [15:18:56.836]
INFO:
prover-client:proving-broker Proving Broker started
15:18:56 [15:18:56.837]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d185a04a0a5a1109 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d185a04a0a5a1109"}
15:18:56 [15:18:56.838]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2X7l3Q/1 with map size 1073741824
15:18:56 [15:18:56.841]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:56 [15:18:56.864]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c8f1b062afe5537 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8c8f1b062afe5537"}
15:18:56 [15:18:56.867]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2X7l3Q/2 with map size 1073741824
15:18:56 [15:18:56.870]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:56 [15:18:56.894]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d185a04a0a5a1109 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d185a04a0a5a1109"}
15:18:56 [15:18:56.895]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d185a04a0a5a1109 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d185a04a0a5a1109"}
15:18:56 [15:18:56.903]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8c8f1b062afe5537 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8c8f1b062afe5537"}
15:18:56 [15:18:56.916]
INFO:
prover-client:proving-broker Proving Broker started
15:18:56 [15:18:56.918]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f55b6114f0b65c22 err=test error
{"provingJoId":"1:BASE_PARITY:f55b6114f0b65c22"}
15:18:56 [15:18:56.921]
INFO:
prover-client:proving-broker Proving Broker started
15:18:56 [15:18:56.922]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fdc1db2ecb330b13 not found
{"provingJobId":"1:BASE_PARITY:fdc1db2ecb330b13"}
15:18:56 [15:18:56.925]
INFO:
prover-client:proving-broker Proving Broker started
15:18:56 [15:18:56.926]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b4fb94803d2e86c epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b4fb94803d2e86c"}
15:18:56 [15:18:56.928]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJeU1tJ/1 with map size 1073741824
15:18:56 [15:18:56.929]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:56 [15:18:56.955]
INFO:
prover-client:proving-broker Proving Broker started
15:18:56 [15:18:56.956]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4687fe93d0f6c89 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e4687fe93d0f6c89"}
15:18:56 [15:18:56.960]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXQkrUK/1 with map size 1073741824
15:18:56 [15:18:56.965]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.083]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e4687fe93d0f6c89 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e4687fe93d0f6c89"}
15:18:57 [15:18:57.101]
INFO:
prover-client:proving-broker Proving Broker started
15:18:57 [15:18:57.102]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12a37595c2611b8f epochNumber=1
{"provingJobId":"1:BASE_PARITY:12a37595c2611b8f"}
15:18:57 [15:18:57.104]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvJ5h7E/1 with map size 1073741824
15:18:57 [15:18:57.107]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.229]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:12a37595c2611b8f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:12a37595c2611b8f"}
15:18:57 [15:18:57.234]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24a7d36562e0a7f0 epochNumber=10
{"provingJobId":"1:BASE_PARITY:24a7d36562e0a7f0"}
15:18:57 [15:18:57.234]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testvJ5h7E/10 with map size 1073741824
15:18:57 [15:18:57.236]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.254]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:57 [15:18:57.255]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:18:57 [15:18:57.384]
INFO:
prover-client:proving-broker Proving Broker started
15:18:57 [15:18:57.386]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f78ab83ecc2c309 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f78ab83ecc2c309"}
15:18:57 [15:18:57.388]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9gu0mD/1 with map size 1073741824
15:18:57 [15:18:57.392]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.587]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8f78ab83ecc2c309 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8f78ab83ecc2c309"}
15:18:57 [15:18:57.627]
INFO:
prover-client:proving-broker Proving Broker started
15:18:57 [15:18:57.628]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:928e21c89cdc0a83 epochNumber=1
{"provingJobId":"1:BASE_PARITY:928e21c89cdc0a83"}
15:18:57 [15:18:57.628]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4mX7OL/1 with map size 1073741824
15:18:57 [15:18:57.632]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.654]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:928e21c89cdc0a83 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:928e21c89cdc0a83"}
15:18:57 [15:18:57.660]
INFO:
prover-client:proving-broker Proving Broker started
15:18:57 [15:18:57.661]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:908d6b7f773bb7e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:908d6b7f773bb7e1"}
15:18:57 [15:18:57.662]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3VP6Ae/1 with map size 1073741824
15:18:57 [15:18:57.667]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.686]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:908d6b7f773bb7e1 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:908d6b7f773bb7e1"}
15:18:57 [15:18:57.687]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:908d6b7f773bb7e1 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:908d6b7f773bb7e1"}
15:18:57 [15:18:57.698]
INFO:
prover-client:proving-broker Proving Broker started
15:18:57 [15:18:57.699]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:476a16e263fb7ab2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:476a16e263fb7ab2"}
15:18:57 [15:18:57.699]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeFFrAJ/1 with map size 1073741824
15:18:57 [15:18:57.703]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.726]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:476a16e263fb7ab2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:476a16e263fb7ab2"}
15:18:57 [15:18:57.739]
INFO:
prover-client:proving-broker Proving Broker started
15:18:57 [15:18:57.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:190e86dc2f218027 epochNumber=1
{"provingJobId":"1:BASE_PARITY:190e86dc2f218027"}
15:18:57 [15:18:57.741]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsUiNe6/1 with map size 1073741824
15:18:57 [15:18:57.743]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.760]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:190e86dc2f218027 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:190e86dc2f218027"}
15:18:57 [15:18:57.761]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b96b024bf899049 epochNumber=3
{"provingJobId":"1:BASE_PARITY:7b96b024bf899049"}
15:18:57 [15:18:57.764]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testsUiNe6/3 with map size 1073741824
15:18:57 [15:18:57.766]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.791]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:57 [15:18:57.792]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:18:57 [15:18:57.810]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:190e86dc2f218027 err=test error
{"provingJoId":"1:BASE_PARITY:190e86dc2f218027"}
15:18:57 [15:18:57.816]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxe6gBb/1 with map size 1073741824
15:18:57 [15:18:57.819]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.844]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testxe6gBb/2 with map size 1073741824
15:18:57 [15:18:57.846]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.866]
INFO:
prover-client:proving-broker Proving Broker started
15:18:57 [15:18:57.873]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1a2d0379ab07bc22 settled=false
{"provingJobId":"1:BASE_PARITY:1a2d0379ab07bc22","status":"pending"}
15:18:57 [15:18:57.875]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:09b72339863c9d67 settled=false
{"provingJobId":"1:BASE_PARITY:09b72339863c9d67","status":"pending"}
15:18:57 [15:18:57.887]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOEuU7c/1 with map size 1073741824
15:18:57 [15:18:57.899]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.925]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOEuU7c/2 with map size 1073741824
15:18:57 [15:18:57.929]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:57 [15:18:57.959]
INFO:
prover-client:proving-broker Proving Broker started
15:18:57 [15:18:57.961]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f6efb0ec82a2af5f settled=true
{"provingJobId":"1:BASE_PARITY:f6efb0ec82a2af5f","status":"fulfilled"}
15:18:57 [15:18:57.963]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:a6968b07213d07e9 settled=true
{"provingJobId":"2:BASE_PARITY:a6968b07213d07e9","status":"fulfilled"}
15:18:57 [15:18:57.968]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwwOeMx/1 with map size 1073741824
15:18:57 [15:18:57.971]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.003]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwwOeMx/2 with map size 1073741824
15:18:58 [15:18:58.012]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.030]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.033]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6aa739b5f5109696 settled=true
{"provingJobId":"1:BASE_PARITY:6aa739b5f5109696","status":"fulfilled"}
15:18:58 [15:18:58.036]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4bac89adc868ddf0 settled=false
{"provingJobId":"1:BASE_PARITY:4bac89adc868ddf0","status":"pending"}
15:18:58 [15:18:58.041]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.041]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:569151680ac6aa49 epochNumber=1
{"provingJobId":"1:BASE_PARITY:569151680ac6aa49"}
15:18:58 [15:18:58.043]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnEzZVe/1 with map size 1073741824
15:18:58 [15:18:58.046]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.072]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f6df2cc199c0358 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f6df2cc199c0358"}
15:18:58 [15:18:58.074]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:6f6df2cc199c0358: Error: db error: Error: db error
15:18:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:6f6df2cc199c0358"}
15:18:58 [15:18:58.079]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fab5611d27da8c61 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fab5611d27da8c61"}
15:18:58 [15:18:58.081]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testn5dAUc/1 with map size 1073741824
15:18:58 [15:18:58.085]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.118]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fab5611d27da8c61 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fab5611d27da8c61"}
15:18:58 [15:18:58.119]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fab5611d27da8c61 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fab5611d27da8c61"}
15:18:58 [15:18:58.133]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c5786435804507e epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c5786435804507e"}
15:18:58 [15:18:58.137]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAeqeTq/1 with map size 1073741824
15:18:58 [15:18:58.140]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.163]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1c5786435804507e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1c5786435804507e"}
15:18:58 [15:18:58.163]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1c5786435804507e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1c5786435804507e"}
15:18:58 [15:18:58.164]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:1c5786435804507e: Error: db error
15:18:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:1c5786435804507e"}
15:18:58 [15:18:58.169]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.169]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3422916fbe7505cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:3422916fbe7505cd"}
15:18:58 [15:18:58.171]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3iQhfM/1 with map size 1073741824
15:18:58 [15:18:58.175]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.200]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3422916fbe7505cd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3422916fbe7505cd"}
15:18:58 [15:18:58.200]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3422916fbe7505cd type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3422916fbe7505cd"}
15:18:58 [15:18:58.216]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.217]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f97ff622b8e11abf epochNumber=1
{"provingJobId":"1:BASE_PARITY:f97ff622b8e11abf"}
15:18:58 [15:18:58.218]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcSmhvt/1 with map size 1073741824
15:18:58 [15:18:58.220]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.243]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f97ff622b8e11abf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f97ff622b8e11abf"}
15:18:58 [15:18:58.244]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f97ff622b8e11abf type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f97ff622b8e11abf"}
15:18:58 [15:18:58.245]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:f97ff622b8e11abf jobErr=test error: Error: db error
15:18:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:f97ff622b8e11abf"}
15:18:58 [15:18:58.249]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.249]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4f58afccfbd2cfaa not found
{"provingJobId":"1:BASE_PARITY:4f58afccfbd2cfaa"}
15:18:58 [15:18:58.255]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.255]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4d81224f044a7cc4 err=test error
{"provingJoId":"1:BASE_PARITY:4d81224f044a7cc4"}
15:18:58 [15:18:58.261]
INFO:
prover-client:proving-broker Proving Broker started
15:18:58 [15:18:58.272]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec976a728cd1c17a epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec976a728cd1c17a"}
15:18:58 [15:18:58.273]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpn29Hd/1 with map size 1073741824
15:18:58 [15:18:58.279]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.306]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ec976a728cd1c17a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ec976a728cd1c17a"}
15:18:58 [15:18:58.307]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ec976a728cd1c17a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ec976a728cd1c17a"}
15:18:58 [15:18:58.325]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:d8bce2905de88a7d epochNumber=2
{"provingJobId":"2:BASE_PARITY:d8bce2905de88a7d"}
15:18:58 [15:18:58.327]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpn29Hd/2 with map size 1073741824
15:18:58 [15:18:58.331]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.354]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:d8bce2905de88a7d type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:d8bce2905de88a7d"}
15:18:58 [15:18:58.354]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:d8bce2905de88a7d type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:d8bce2905de88a7d"}
15:18:58 [15:18:58.426]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:ed7a9e0d6e17fdf1 epochNumber=3
{"provingJobId":"3:BASE_PARITY:ed7a9e0d6e17fdf1"}
15:18:58 [15:18:58.429]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testpn29Hd/3 with map size 1073741824
15:18:58 [15:18:58.437]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.466]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:58 [15:18:58.468]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:18:58 [15:18:58.524]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:de861173c51f71a0 epochNumber=4
{"provingJobId":"4:BASE_PARITY:de861173c51f71a0"}
15:18:58 [15:18:58.525]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testpn29Hd/4 with map size 1073741824
15:18:58 [15:18:58.530]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.578]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:58 [15:18:58.579]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
15:18:58 [15:18:58.628]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:3a543273c9f82aab epochNumber=5
{"provingJobId":"5:BASE_PARITY:3a543273c9f82aab"}
15:18:58 [15:18:58.629]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testpn29Hd/5 with map size 1073741824
15:18:58 [15:18:58.632]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:18:58 [15:18:58.656]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:18:58 [15:18:58.657]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
15:18:58
PASS src/proving_broker/proving_broker.test.ts (
13.55 s)
15:18:58 ProvingBroker
15:18:58 Producer API
15:18:58
✓ refuses stale jobs (52 ms)
15:18:58
✓ enqueues jobs (5 ms)
15:18:58
✓ ignores duplicate jobs (4 ms)
15:18:58
✓ reports correct status when enqueuing repeat jobs (5 ms)
15:18:58
✓ throws an error in case of duplicate job IDs (36 ms)
15:18:58
✓ returns not-found status for non-existing jobs (2 ms)
15:18:58
✓ cancels jobs in queue (4 ms)
15:18:58
✓ cancels jobs in-progress (3 ms)
15:18:58
✓ returns job result if successful (10 ms)
15:18:58
✓ returns job error if failed (3 ms)
15:18:58
✓ correctly returns job status for concurrent writes (3 ms)
15:18:58
✓ refuses stale jobs (40 ms)
15:18:58
✓ enqueues jobs (43 ms)
15:18:58
✓ ignores duplicate jobs (25 ms)
15:18:58
✓ reports correct status when enqueuing repeat jobs (31 ms)
15:18:58
✓ throws an error in case of duplicate job IDs (38 ms)
15:18:58
✓ returns not-found status for non-existing jobs (3 ms)
15:18:58
✓ cancels jobs in queue (29 ms)
15:18:58
✓ cancels jobs in-progress (33 ms)
15:18:58
✓ returns job result if successful (45 ms)
15:18:58
✓ returns job error if failed (35 ms)
15:18:58
✓ correctly returns job status for concurrent writes (41 ms)
15:18:58 Consumer API
15:18:58
✓ returns undefined if no jobs are available (1 ms)
15:18:58
✓ returns jobs in priority order (2 ms)
15:18:58
✓ returns undefined if no jobs are available for the given allowList (1 ms)
15:18:58
✓ returns a job if it is in the allowList (3 ms)
15:18:58
✓ returns the most important job if it is in the allowList (2 ms)
15:18:58
✓ returns any job if filter is empty (2 ms)
15:18:58
✓ returns a new job when reporting job success (2 ms)
15:18:58
✓ returns a new job when reporting permanent error (2 ms)
15:18:58
✓ returns a new job when reporting retry-able error (2 ms)
15:18:58
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
15:18:58
✓ returns a new job if job is already in progress elsewhere (1055 ms)
15:18:58
✓ avoids sending the same job to a new agent after a restart (1005 ms)
15:18:58
✓ avoids sending a completed job to a new agent after a restart (1009 ms)
15:18:58
✓ tracks job result if in progress (4 ms)
15:18:58
✓ tracks job result even if job is in queue (4 ms)
15:18:58
✓ ignores reported job error if unknown job (2 ms)
15:18:58
✓ ignores job result if unknown job (2 ms)
15:18:58
✓ returns undefined if no jobs are available (3 ms)
15:18:58
✓ returns jobs in priority order (51 ms)
15:18:58
✓ returns undefined if no jobs are available for the given allowList (24 ms)
15:18:58
✓ returns a job if it is in the allowList (63 ms)
15:18:58
✓ returns the most important job if it is in the allowList (59 ms)
15:18:58
✓ returns any job if filter is empty (66 ms)
15:18:58
✓ returns a new job when reporting job success (45 ms)
15:18:58
✓ returns a new job when reporting permanent error (36 ms)
15:18:58
✓ returns a new job when reporting retry-able error (33 ms)
15:18:58
✓ returns a new job when reporting progress if current one is cancelled (51 ms)
15:18:58
✓ returns a new job if job is already in progress elsewhere (1107 ms)
15:18:58
✓ avoids sending the same job to a new agent after a restart (1063 ms)
15:18:58
✓ avoids sending a completed job to a new agent after a restart (1058 ms)
15:18:58
✓ tracks job result if in progress (70 ms)
15:18:58
✓ tracks job result even if job is in queue (78 ms)
15:18:58
✓ ignores reported job error if unknown job (5 ms)
15:18:58
✓ ignores job result if unknown job (4 ms)
15:18:58 Timeouts
15:18:58
✓ tracks in progress jobs (3 ms)
15:18:58
✓ re-enqueues jobs that time out (110 ms)
15:18:58
✓ cancel stale jobs that time out (229 ms)
15:18:58
✓ keeps the jobs in progress while it is alive (207 ms)
15:18:58
✓ tracks in progress jobs (29 ms)
15:18:58
✓ re-enqueues jobs that time out (140 ms)
15:18:58
✓ cancel stale jobs that time out (288 ms)
15:18:58
✓ keeps the jobs in progress while it is alive (242 ms)
15:18:58 Retries
15:18:58
✓ retries jobs (2 ms)
15:18:58
✓ retries up to a maximum number of times (3 ms)
15:18:58
✓ passing retry=false does not retry (2 ms)
15:18:58
✓ does not retry if job is stale (29 ms)
15:18:58
✓ retries jobs (32 ms)
15:18:58
✓ retries up to a maximum number of times (38 ms)
15:18:58
✓ passing retry=false does not retry (39 ms)
15:18:58
✓ does not retry if job is stale (78 ms)
15:18:58 Database management
15:18:58
✓ re-enqueues proof requests on start (3 ms)
15:18:58
✓ restores proof results on start (1 ms)
15:18:58
✓ only re-enqueues unfinished jobs (3 ms)
15:18:58
✓ saves job when enqueued (2 ms)
15:18:58
✓ does not retain job if database fails to save (2 ms)
15:18:58
✓ saves job result (2 ms)
15:18:58
✓ saves result even if database fails to save (2 ms)
15:18:58
✓ saves job error (2 ms)
15:18:58
✓ saves job error even if database fails to save (3 ms)
15:18:58
✓ does not save job result if job is unknown (1 ms)
15:18:58
✓ does not save job error if job is unknown (2 ms)
15:18:58
✓ cleans up old jobs periodically (266 ms)
15:18:58
✓ re-enqueues proof requests on start (66 ms)
15:18:58
✓ restores proof results on start (86 ms)
15:18:58
✓ only re-enqueues unfinished jobs (73 ms)
15:18:58
✓ saves job when enqueued (31 ms)
15:18:58
✓ does not retain job if database fails to save (4 ms)
15:18:58
✓ saves job result (55 ms)
15:18:58
✓ saves result even if database fails to save (35 ms)
15:18:58
✓ saves job error (45 ms)
15:18:58
✓ saves job error even if database fails to save (34 ms)
15:18:58
✓ does not save job result if job is unknown (3 ms)
15:18:58
✓ does not save job error if job is unknown (6 ms)
15:18:58
✓ cleans up old jobs periodically (453 ms)
15:18:58
15:18:58
Test Suites: 1 passed, 1 total
15:18:58
Tests: 96 passed, 96 total
15:18:58
Snapshots: 0 total
15:18:58
Time: 13.667 s
15:18:58
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
15:18:58
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?