Parent Log:
http://ci.aztec-labs.com/844bec8772ac4116
Command: fa3d426c5ffed922 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/ddabb45f79cc171e41a0a5330544775de9fdf7fd
Env: REF_NAME=gh-readonly-queue/next/pr-14983-4e03f15ca8099f25d50eddd19c4914764b84fe7b CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 13 14:38:03 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14983_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
14:38:07 [14:38:07.328]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:39f195e5be2a7595 epochNumber=42
{"provingJobId":"1:BASE_PARITY:39f195e5be2a7595"}
14:38:07 [14:38:07.338]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:417a6b1104b113e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:417a6b1104b113e7"}
14:38:07 [14:38:07.389]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.390]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c84725233cfe868 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c84725233cfe868"}
14:38:07 [14:38:07.391]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75d9efb17ce7eefc epochNumber=1
{"provingJobId":"1:BASE_PARITY:75d9efb17ce7eefc"}
14:38:07 [14:38:07.392]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.393]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0749f5381a480d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c0749f5381a480d6"}
14:38:07 [14:38:07.394]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c0749f5381a480d6 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c0749f5381a480d6"}
14:38:07 [14:38:07.396]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.397]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9cae476ac88ae29e epochNumber=1
{"provingJobId":"1:BASE_PARITY:9cae476ac88ae29e"}
14:38:07 [14:38:07.398]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9cae476ac88ae29e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9cae476ac88ae29e"}
14:38:07 [14:38:07.399]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9cae476ac88ae29e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9cae476ac88ae29e"}
14:38:07 [14:38:07.400]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9cae476ac88ae29e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9cae476ac88ae29e"}
14:38:07 [14:38:07.401]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.401]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:008fca486f8c269c epochNumber=1
{"provingJobId":"1:BASE_PARITY:008fca486f8c269c"}
14:38:07 [14:38:07.428]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.429]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.430]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3dbc795ef20845e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3dbc795ef20845e8"}
14:38:07 [14:38:07.431]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3dbc795ef20845e8
{"provingJobId":"1:BASE_PARITY:3dbc795ef20845e8"}
14:38:07 [14:38:07.431]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3dbc795ef20845e8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3dbc795ef20845e8"}
14:38:07 [14:38:07.431]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3dbc795ef20845e8 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3dbc795ef20845e8"}
14:38:07 [14:38:07.433]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.434]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c752abecdf3d5475 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c752abecdf3d5475"}
14:38:07 [14:38:07.434]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c752abecdf3d5475
{"provingJobId":"1:BASE_PARITY:c752abecdf3d5475"}
14:38:07 [14:38:07.435]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c752abecdf3d5475 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c752abecdf3d5475"}
14:38:07 [14:38:07.436]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:181335025647b153 epochNumber=1
{"provingJobId":"1:BASE_PARITY:181335025647b153"}
14:38:07 [14:38:07.438]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:181335025647b153 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:181335025647b153"}
14:38:07 [14:38:07.438]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:181335025647b153 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:181335025647b153"}
14:38:07 [14:38:07.440]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.440]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e13ecff2d061239b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e13ecff2d061239b"}
14:38:07 [14:38:07.441]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e13ecff2d061239b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e13ecff2d061239b"}
14:38:07 [14:38:07.441]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e13ecff2d061239b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e13ecff2d061239b"}
14:38:07 [14:38:07.442]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.443]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5780c3ebb3a4d48e epochNumber=0
{"provingJobId":"1:BASE_PARITY:5780c3ebb3a4d48e"}
14:38:07 [14:38:07.443]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5780c3ebb3a4d48e epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5780c3ebb3a4d48e"}
14:38:07 [14:38:07.443]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c11bd47980e9dd16 epochNumber=0
{"provingJobId":"1:BASE_PARITY:c11bd47980e9dd16"}
14:38:07 [14:38:07.443]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5780c3ebb3a4d48e epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5780c3ebb3a4d48e"}
14:38:07 [14:38:07.444]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f19962d17a54d45 epochNumber=0
{"provingJobId":"1:BASE_PARITY:7f19962d17a54d45"}
14:38:07 [14:38:07.445]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.446]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.446]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36562d92b6b057da epochNumber=2
{"provingJobId":"1:BASE_PARITY:36562d92b6b057da"}
14:38:07 [14:38:07.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e2739e53b14c862 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e2739e53b14c862"}
14:38:07 [14:38:07.448]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.448]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2012503c06f81ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:c2012503c06f81ed"}
14:38:07 [14:38:07.449]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.449]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a81c44e7706b547f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a81c44e7706b547f"}
14:38:07 [14:38:07.450]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a14fbdc27f39f548 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a14fbdc27f39f548"}
14:38:07 [14:38:07.450]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2216f3fd5cf1551 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f2216f3fd5cf1551"}
14:38:07 [14:38:07.450]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc3fa8db76872045 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc3fa8db76872045"}
14:38:07 [14:38:07.451]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:771bc6fe96b323c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:771bc6fe96b323c3"}
14:38:07 [14:38:07.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ae2add7dca17df4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ae2add7dca17df4"}
14:38:07 [14:38:07.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10223c2038f1e521 epochNumber=2
{"provingJobId":"1:BASE_PARITY:10223c2038f1e521"}
14:38:07 [14:38:07.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2efd7cc9babec23 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2efd7cc9babec23"}
14:38:07 [14:38:07.454]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.454]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9bfc103fc2abe89c epochNumber=1
{"provingJobId":"1:BASE_PARITY:9bfc103fc2abe89c"}
14:38:07 [14:38:07.454]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:62829bbfa4375706 epochNumber=1
{"provingJobId":"1:BASE_PARITY:62829bbfa4375706"}
14:38:07 [14:38:07.455]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0021a5aef4bef78c epochNumber=2
{"provingJobId":"1:BASE_PARITY:0021a5aef4bef78c"}
14:38:07 [14:38:07.455]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e9e84d27cb2c4ae epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e9e84d27cb2c4ae"}
14:38:07 [14:38:07.470]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.471]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:49b05c9cf0190c6d epochNumber=1
{"provingJobId":"1:BASE_PARITY:49b05c9cf0190c6d"}
14:38:07 [14:38:07.472]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0bbd48dd5e48cd16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0bbd48dd5e48cd16"}
14:38:07 [14:38:07.473]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:49b05c9cf0190c6d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:49b05c9cf0190c6d"}
14:38:07 [14:38:07.477]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:33ace15947265fa0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:33ace15947265fa0"}
14:38:07 [14:38:07.479]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfa6bb6c93aa51de epochNumber=1
{"provingJobId":"1:BASE_PARITY:dfa6bb6c93aa51de"}
14:38:07 [14:38:07.479]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:33ace15947265fa0 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:33ace15947265fa0"}
14:38:07 [14:38:07.480]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.481]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b4a368773906e33 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b4a368773906e33"}
14:38:07 [14:38:07.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a9c2add61439838 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a9c2add61439838"}
14:38:07 [14:38:07.482]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0b4a368773906e33 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:0b4a368773906e33"}
14:38:07 [14:38:07.483]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02d4ac6ba3329898 epochNumber=1
{"provingJobId":"1:BASE_PARITY:02d4ac6ba3329898"}
14:38:07 [14:38:07.484]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:02d4ac6ba3329898
{"provingJobId":"1:BASE_PARITY:02d4ac6ba3329898"}
14:38:07 [14:38:07.485]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:02d4ac6ba3329898 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:02d4ac6ba3329898"}
14:38:07 [14:38:07.485]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b8382c6fa73e9c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b8382c6fa73e9c9"}
14:38:07 [14:38:07.485]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:02d4ac6ba3329898 has already been completed
{"provingJobId":"1:BASE_PARITY:02d4ac6ba3329898"}
14:38:07 [14:38:07.487]
INFO:
prover-client:proving-broker Proving Broker started
14:38:07 [14:38:07.487]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d161934e3657947b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d161934e3657947b"}
14:38:07 [14:38:07.487]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34ab899ae541540f epochNumber=2
{"provingJobId":"1:BASE_PARITY:34ab899ae541540f"}
14:38:08 [14:38:08.540]
INFO:
prover-client:proving-broker Proving Broker started
14:38:08 [14:38:08.541]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d161934e3657947b settled=false
{"provingJobId":"1:BASE_PARITY:d161934e3657947b","status":"pending"}
14:38:08 [14:38:08.542]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:34ab899ae541540f settled=false
{"provingJobId":"1:BASE_PARITY:34ab899ae541540f","status":"pending"}
14:38:08 [14:38:08.543]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d161934e3657947b type=BASE_PARITY startedAt=1749825487487 older agent has taken job
{"provingJobId":"1:BASE_PARITY:d161934e3657947b"}
14:38:08 [14:38:08.544]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d161934e3657947b type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:d161934e3657947b"}
14:38:08 [14:38:08.545]
INFO:
prover-client:proving-broker Proving Broker started
14:38:08 [14:38:08.546]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dee500c7e932f453 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dee500c7e932f453"}
14:38:08 [14:38:08.546]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df541d193eda4e92 epochNumber=2
{"provingJobId":"1:BASE_PARITY:df541d193eda4e92"}
14:38:09 [14:38:09.546]
INFO:
prover-client:proving-broker Proving Broker started
14:38:09 [14:38:09.547]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dee500c7e932f453 settled=false
{"provingJobId":"1:BASE_PARITY:dee500c7e932f453","status":"pending"}
14:38:09 [14:38:09.547]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:df541d193eda4e92 settled=false
{"provingJobId":"1:BASE_PARITY:df541d193eda4e92","status":"pending"}
14:38:09 [14:38:09.548]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dee500c7e932f453 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:dee500c7e932f453"}
14:38:09 [14:38:09.549]
INFO:
prover-client:proving-broker Proving Broker started
14:38:09 [14:38:09.550]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a094adcd9f3695a9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a094adcd9f3695a9"}
14:38:09 [14:38:09.550]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bcd79fd403306038 epochNumber=2
{"provingJobId":"1:BASE_PARITY:bcd79fd403306038"}
14:38:10 [14:38:10.551]
INFO:
prover-client:proving-broker Proving Broker started
14:38:10 [14:38:10.551]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a094adcd9f3695a9 settled=false
{"provingJobId":"1:BASE_PARITY:a094adcd9f3695a9","status":"pending"}
14:38:10 [14:38:10.552]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bcd79fd403306038 settled=false
{"provingJobId":"1:BASE_PARITY:bcd79fd403306038","status":"pending"}
14:38:10 [14:38:10.552]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a094adcd9f3695a9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a094adcd9f3695a9"}
14:38:10 [14:38:10.553]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a094adcd9f3695a9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a094adcd9f3695a9"}
14:38:10 [14:38:10.554]
INFO:
prover-client:proving-broker Proving Broker started
14:38:10 [14:38:10.555]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6eaed8ee4556d95 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6eaed8ee4556d95"}
14:38:10 [14:38:10.555]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:522ea5253b33881b epochNumber=2
{"provingJobId":"1:BASE_PARITY:522ea5253b33881b"}
14:38:10 [14:38:10.556]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a6eaed8ee4556d95 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a6eaed8ee4556d95"}
14:38:10 [14:38:10.557]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:522ea5253b33881b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:522ea5253b33881b"}
14:38:10 [14:38:10.558]
INFO:
prover-client:proving-broker Proving Broker started
14:38:10 [14:38:10.558]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5790dae70703a176 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5790dae70703a176"}
14:38:10 [14:38:10.559]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f1feb4edd115a2dd epochNumber=2
{"provingJobId":"1:BASE_PARITY:f1feb4edd115a2dd"}
14:38:10 [14:38:10.559]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5790dae70703a176 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5790dae70703a176"}
14:38:10 [14:38:10.559]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5790dae70703a176 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5790dae70703a176"}
14:38:10 [14:38:10.560]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f1feb4edd115a2dd type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f1feb4edd115a2dd"}
14:38:10 [14:38:10.561]
INFO:
prover-client:proving-broker Proving Broker started
14:38:10 [14:38:10.561]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:12be39b39565bc32 err=test error
{"provingJoId":"1:BASE_PARITY:12be39b39565bc32"}
14:38:10 [14:38:10.562]
INFO:
prover-client:proving-broker Proving Broker started
14:38:10 [14:38:10.563]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2dbabe742666bcdf not found
{"provingJobId":"1:BASE_PARITY:2dbabe742666bcdf"}
14:38:10 [14:38:10.564]
INFO:
prover-client:proving-broker Proving Broker started
14:38:10 [14:38:10.566]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ffe8354d833e8ef4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ffe8354d833e8ef4"}
14:38:10 [14:38:10.567]
INFO:
prover-client:proving-broker Proving Broker started
14:38:10 [14:38:10.568]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aeef1d5b92002711 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aeef1d5b92002711"}
14:38:10 [14:38:10.670]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aeef1d5b92002711 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:aeef1d5b92002711"}
14:38:10 [14:38:10.678]
INFO:
prover-client:proving-broker Proving Broker started
14:38:10 [14:38:10.679]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2add9bfa889dd30 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e2add9bfa889dd30"}
14:38:10 [14:38:10.780]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e2add9bfa889dd30 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e2add9bfa889dd30"}
14:38:10 [14:38:10.781]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd0f026a2c53251a epochNumber=10
{"provingJobId":"1:BASE_PARITY:dd0f026a2c53251a"}
14:38:10 [14:38:10.805]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:10 [14:38:10.907]
INFO:
prover-client:proving-broker Proving Broker started
14:38:10 [14:38:10.908]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a245e8856c5ee7ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:a245e8856c5ee7ba"}
14:38:11 [14:38:11.083]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a245e8856c5ee7ba timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a245e8856c5ee7ba"}
14:38:11 [14:38:11.111]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.112]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e2b8e4317231451 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e2b8e4317231451"}
14:38:11 [14:38:11.113]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0e2b8e4317231451 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0e2b8e4317231451"}
14:38:11 [14:38:11.113]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.114]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8940c6f5598b0c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8940c6f5598b0c5"}
14:38:11 [14:38:11.114]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b8940c6f5598b0c5 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b8940c6f5598b0c5"}
14:38:11 [14:38:11.115]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b8940c6f5598b0c5 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:b8940c6f5598b0c5"}
14:38:11 [14:38:11.116]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.116]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7793a25b5f7ae639 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7793a25b5f7ae639"}
14:38:11 [14:38:11.116]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7793a25b5f7ae639 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7793a25b5f7ae639"}
14:38:11 [14:38:11.117]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.118]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ce5e0a43050c437 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1ce5e0a43050c437"}
14:38:11 [14:38:11.118]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1ce5e0a43050c437 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1ce5e0a43050c437"}
14:38:11 [14:38:11.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:74a0cd374a17737d epochNumber=3
{"provingJobId":"1:BASE_PARITY:74a0cd374a17737d"}
14:38:11 [14:38:11.143]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:11 [14:38:11.145]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1ce5e0a43050c437 err=test error
{"provingJoId":"1:BASE_PARITY:1ce5e0a43050c437"}
14:38:11 [14:38:11.146]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.147]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:01cd0dd06c906c3b settled=false
{"provingJobId":"1:BASE_PARITY:01cd0dd06c906c3b","status":"pending"}
14:38:11 [14:38:11.147]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3f3e652099d2a889 settled=false
{"provingJobId":"1:BASE_PARITY:3f3e652099d2a889","status":"pending"}
14:38:11 [14:38:11.149]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.149]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3ace7ff6cde916e6 settled=true
{"provingJobId":"1:BASE_PARITY:3ace7ff6cde916e6","status":"fulfilled"}
14:38:11 [14:38:11.149]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:e4b004ac53a85c4d settled=true
{"provingJobId":"2:BASE_PARITY:e4b004ac53a85c4d","status":"fulfilled"}
14:38:11 [14:38:11.151]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.151]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7f4f9eb140d510d0 settled=true
{"provingJobId":"1:BASE_PARITY:7f4f9eb140d510d0","status":"fulfilled"}
14:38:11 [14:38:11.151]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0bdb5e121226ae52 settled=false
{"provingJobId":"1:BASE_PARITY:0bdb5e121226ae52","status":"pending"}
14:38:11 [14:38:11.152]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.153]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:29f949d9d2a96991 epochNumber=1
{"provingJobId":"1:BASE_PARITY:29f949d9d2a96991"}
14:38:11 [14:38:11.155]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.155]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:417d1a7ca8575458 epochNumber=1
{"provingJobId":"1:BASE_PARITY:417d1a7ca8575458"}
14:38:11 [14:38:11.156]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:417d1a7ca8575458: Error: db error: Error: db error
14:38:11 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:417d1a7ca8575458"}
14:38:11 [14:38:11.157]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.157]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66e379de2149a82f epochNumber=1
{"provingJobId":"1:BASE_PARITY:66e379de2149a82f"}
14:38:11 [14:38:11.158]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:66e379de2149a82f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:66e379de2149a82f"}
14:38:11 [14:38:11.158]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:66e379de2149a82f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:66e379de2149a82f"}
14:38:11 [14:38:11.159]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19d06bb6ef18970e epochNumber=1
{"provingJobId":"1:BASE_PARITY:19d06bb6ef18970e"}
14:38:11 [14:38:11.160]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:19d06bb6ef18970e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:19d06bb6ef18970e"}
14:38:11 [14:38:11.160]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:19d06bb6ef18970e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:19d06bb6ef18970e"}
14:38:11 [14:38:11.160]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:19d06bb6ef18970e: Error: db error
14:38:11 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:19d06bb6ef18970e"}
14:38:11 [14:38:11.161]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.162]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e06698e75923783 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e06698e75923783"}
14:38:11 [14:38:11.162]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7e06698e75923783 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7e06698e75923783"}
14:38:11 [14:38:11.162]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7e06698e75923783 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7e06698e75923783"}
14:38:11 [14:38:11.167]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58391d01b0743fba epochNumber=1
{"provingJobId":"1:BASE_PARITY:58391d01b0743fba"}
14:38:11 [14:38:11.168]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:58391d01b0743fba type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:58391d01b0743fba"}
14:38:11 [14:38:11.168]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:58391d01b0743fba type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:58391d01b0743fba"}
14:38:11 [14:38:11.169]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:58391d01b0743fba jobErr=test error: Error: db error
14:38:11 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:58391d01b0743fba"}
14:38:11 [14:38:11.170]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.171]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:624138071b648e4a not found
{"provingJobId":"1:BASE_PARITY:624138071b648e4a"}
14:38:11 [14:38:11.172]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.173]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7df369328975513d err=test error
{"provingJoId":"1:BASE_PARITY:7df369328975513d"}
14:38:11 [14:38:11.174]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.184]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d9f2bb8d0c32542 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d9f2bb8d0c32542"}
14:38:11 [14:38:11.185]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0d9f2bb8d0c32542 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0d9f2bb8d0c32542"}
14:38:11 [14:38:11.185]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0d9f2bb8d0c32542 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0d9f2bb8d0c32542"}
14:38:11 [14:38:11.195]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:83aeff5a6f03b913 epochNumber=2
{"provingJobId":"2:BASE_PARITY:83aeff5a6f03b913"}
14:38:11 [14:38:11.195]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:83aeff5a6f03b913 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:83aeff5a6f03b913"}
14:38:11 [14:38:11.196]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:83aeff5a6f03b913 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:83aeff5a6f03b913"}
14:38:11 [14:38:11.257]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:6c0e292519910bb6 epochNumber=3
{"provingJobId":"3:BASE_PARITY:6c0e292519910bb6"}
14:38:11 [14:38:11.275]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:11 [14:38:11.319]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:10e1ce6461074e5c epochNumber=4
{"provingJobId":"4:BASE_PARITY:10e1ce6461074e5c"}
14:38:11 [14:38:11.326]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:11 [14:38:11.381]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:7275723796c638e7 epochNumber=5
{"provingJobId":"5:BASE_PARITY:7275723796c638e7"}
14:38:11 [14:38:11.404]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:11 [14:38:11.435]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.436]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15232034a1259973 epochNumber=42
{"provingJobId":"1:BASE_PARITY:15232034a1259973"}
14:38:11 [14:38:11.437]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testgNZVRP/42 with map size 1073741824
14:38:11 [14:38:11.441]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.466]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:6fff90d14a174b9a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6fff90d14a174b9a"}
14:38:11 [14:38:11.470]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.471]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e6d84771256945a epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e6d84771256945a"}
14:38:11 [14:38:11.472]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testahITYG/1 with map size 1073741824
14:38:11 [14:38:11.473]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.494]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8aa82fecba4c6027 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8aa82fecba4c6027"}
14:38:11 [14:38:11.506]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.507]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97075ee7dfef9a26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:97075ee7dfef9a26"}
14:38:11 [14:38:11.508]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwywkKY/1 with map size 1073741824
14:38:11 [14:38:11.512]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.532]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:97075ee7dfef9a26 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:97075ee7dfef9a26"}
14:38:11 [14:38:11.535]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.536]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c451580a4f3e4005 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c451580a4f3e4005"}
14:38:11 [14:38:11.537]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaZPuDF/1 with map size 1073741824
14:38:11 [14:38:11.539]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.557]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c451580a4f3e4005 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c451580a4f3e4005"}
14:38:11 [14:38:11.558]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c451580a4f3e4005 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c451580a4f3e4005"}
14:38:11 [14:38:11.564]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c451580a4f3e4005 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c451580a4f3e4005"}
14:38:11 [14:38:11.568]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.569]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8650e1da4e6fea7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8650e1da4e6fea7"}
14:38:11 [14:38:11.570]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLqPHpw/1 with map size 1073741824
14:38:11 [14:38:11.572]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.596]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.599]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.600]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67106d5831cbc974 epochNumber=1
{"provingJobId":"1:BASE_PARITY:67106d5831cbc974"}
14:38:11 [14:38:11.601]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAxtZjt/1 with map size 1073741824
14:38:11 [14:38:11.602]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.620]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:67106d5831cbc974
{"provingJobId":"1:BASE_PARITY:67106d5831cbc974"}
14:38:11 [14:38:11.621]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:67106d5831cbc974 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:67106d5831cbc974"}
14:38:11 [14:38:11.621]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:67106d5831cbc974 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:67106d5831cbc974"}
14:38:11 [14:38:11.630]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.631]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:37f07fed9ce48b8b epochNumber=1
{"provingJobId":"1:BASE_PARITY:37f07fed9ce48b8b"}
14:38:11 [14:38:11.631]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3veglo/1 with map size 1073741824
14:38:11 [14:38:11.634]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.653]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:37f07fed9ce48b8b
{"provingJobId":"1:BASE_PARITY:37f07fed9ce48b8b"}
14:38:11 [14:38:11.653]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:37f07fed9ce48b8b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:37f07fed9ce48b8b"}
14:38:11 [14:38:11.663]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.664]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5014b5fd69aaf89 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5014b5fd69aaf89"}
14:38:11 [14:38:11.665]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testG2yIjV/1 with map size 1073741824
14:38:11 [14:38:11.666]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.683]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b5014b5fd69aaf89 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b5014b5fd69aaf89"}
14:38:11 [14:38:11.684]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b5014b5fd69aaf89 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b5014b5fd69aaf89"}
14:38:11 [14:38:11.692]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a4fa9baa437e245a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a4fa9baa437e245a"}
14:38:11 [14:38:11.693]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testztiyrD/1 with map size 1073741824
14:38:11 [14:38:11.696]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.712]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a4fa9baa437e245a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a4fa9baa437e245a"}
14:38:11 [14:38:11.712]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a4fa9baa437e245a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a4fa9baa437e245a"}
14:38:11 [14:38:11.720]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.721]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d49c7d793a357527 epochNumber=0
{"provingJobId":"1:BASE_PARITY:d49c7d793a357527"}
14:38:11 [14:38:11.721]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test0rhQAF/0 with map size 1073741824
14:38:11 [14:38:11.723]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.740]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d49c7d793a357527 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d49c7d793a357527"}
14:38:11 [14:38:11.741]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:49774678e311455a epochNumber=0
{"provingJobId":"1:BASE_PARITY:49774678e311455a"}
14:38:11 [14:38:11.741]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d49c7d793a357527 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d49c7d793a357527"}
14:38:11 [14:38:11.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ead1fea1d9893aee epochNumber=0
{"provingJobId":"1:BASE_PARITY:ead1fea1d9893aee"}
14:38:11 [14:38:11.756]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.761]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0236983f3934db4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e0236983f3934db4"}
14:38:11 [14:38:11.762]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0wEtlq/2 with map size 1073741824
14:38:11 [14:38:11.764]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86518e6356493e4f epochNumber=1
{"provingJobId":"1:BASE_PARITY:86518e6356493e4f"}
14:38:11 [14:38:11.783]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0wEtlq/1 with map size 1073741824
14:38:11 [14:38:11.785]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.803]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.803]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:372117b0f26cf984 epochNumber=1
{"provingJobId":"1:BASE_PARITY:372117b0f26cf984"}
14:38:11 [14:38:11.804]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrR5uhz/1 with map size 1073741824
14:38:11 [14:38:11.807]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.823]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.824]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a2f5109b87770992 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a2f5109b87770992"}
14:38:11 [14:38:11.824]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9VYiRD/1 with map size 1073741824
14:38:11 [14:38:11.826]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.841]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03b2bdfc7b67aca5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:03b2bdfc7b67aca5"}
14:38:11 [14:38:11.866]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc30712e4686b713 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cc30712e4686b713"}
14:38:11 [14:38:11.868]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test9VYiRD/2 with map size 1073741824
14:38:11 [14:38:11.870]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.889]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67146da1dcefcb39 epochNumber=1
{"provingJobId":"1:BASE_PARITY:67146da1dcefcb39"}
14:38:11 [14:38:11.900]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.901]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea6aa10b65c4fd34 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea6aa10b65c4fd34"}
14:38:11 [14:38:11.901]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxsLcss/1 with map size 1073741824
14:38:11 [14:38:11.904]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.926]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:07d9f86a0cb47a98 epochNumber=1
{"provingJobId":"1:BASE_PARITY:07d9f86a0cb47a98"}
14:38:11 [14:38:11.933]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eadd733fea38343e epochNumber=2
{"provingJobId":"1:BASE_PARITY:eadd733fea38343e"}
14:38:11 [14:38:11.934]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testxsLcss/2 with map size 1073741824
14:38:11 [14:38:11.936]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.953]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a14f0d2e1c2d6ce1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a14f0d2e1c2d6ce1"}
14:38:11 [14:38:11.964]
INFO:
prover-client:proving-broker Proving Broker started
14:38:11 [14:38:11.965]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47fa9a9e5dd56502 epochNumber=1
{"provingJobId":"1:BASE_PARITY:47fa9a9e5dd56502"}
14:38:11 [14:38:11.966]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrKsIbP/1 with map size 1073741824
14:38:11 [14:38:11.969]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:11 [14:38:11.989]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d1b64b5449de011 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d1b64b5449de011"}
14:38:11 [14:38:11.995]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82a3733283680972 epochNumber=2
{"provingJobId":"1:BASE_PARITY:82a3733283680972"}
14:38:11 [14:38:11.996]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrKsIbP/2 with map size 1073741824
14:38:11 [14:38:11.998]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:12 [14:38:12.018]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9dcf13422671427 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c9dcf13422671427"}
14:38:12 [14:38:12.029]
INFO:
prover-client:proving-broker Proving Broker started
14:38:12 [14:38:12.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2799ccba59b6ae69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2799ccba59b6ae69"}
14:38:12 [14:38:12.030]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjaTvTw/1 with map size 1073741824
14:38:12 [14:38:12.033]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:12 [14:38:12.049]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:810ea4b51280a94a epochNumber=1
{"provingJobId":"1:BASE_PARITY:810ea4b51280a94a"}
14:38:12 [14:38:12.055]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2799ccba59b6ae69 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2799ccba59b6ae69"}
14:38:12 [14:38:12.065]
INFO:
prover-client:proving-broker Proving Broker started
14:38:12 [14:38:12.065]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6bec9660f9c7e7a epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6bec9660f9c7e7a"}
14:38:12 [14:38:12.066]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testj3Vu18/1 with map size 1073741824
14:38:12 [14:38:12.069]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:12 [14:38:12.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2e4342022cc7a03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e2e4342022cc7a03"}
14:38:12 [14:38:12.092]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c6bec9660f9c7e7a type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:c6bec9660f9c7e7a"}
14:38:12 [14:38:12.105]
INFO:
prover-client:proving-broker Proving Broker started
14:38:12 [14:38:12.106]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1893c5f4113573e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c1893c5f4113573e"}
14:38:12 [14:38:12.106]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjpvff1/1 with map size 1073741824
14:38:12 [14:38:12.108]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:12 [14:38:12.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b64926beb1d5fbc7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b64926beb1d5fbc7"}
14:38:12 [14:38:12.135]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c1893c5f4113573e type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:c1893c5f4113573e"}
14:38:12 [14:38:12.139]
INFO:
prover-client:proving-broker Proving Broker started
14:38:12 [14:38:12.140]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:540da426f404c519 epochNumber=1
{"provingJobId":"1:BASE_PARITY:540da426f404c519"}
14:38:12 [14:38:12.141]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxUnyCa/1 with map size 1073741824
14:38:12 [14:38:12.144]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:12 [14:38:12.167]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:540da426f404c519
{"provingJobId":"1:BASE_PARITY:540da426f404c519"}
14:38:12 [14:38:12.167]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:540da426f404c519 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:540da426f404c519"}
14:38:12 [14:38:12.172]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f270ca5a16cdde6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f270ca5a16cdde6"}
14:38:12 [14:38:12.180]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:540da426f404c519 has already been completed
{"provingJobId":"1:BASE_PARITY:540da426f404c519"}
14:38:12 [14:38:12.186]
INFO:
prover-client:proving-broker Proving Broker started
14:38:12 [14:38:12.187]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a769f8f8161d3fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a769f8f8161d3fd"}
14:38:12 [14:38:12.189]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQcKfzc/1 with map size 1073741824
14:38:12 [14:38:12.192]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:12 [14:38:12.213]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ff10eb3b832eff4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6ff10eb3b832eff4"}
14:38:12 [14:38:12.215]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQcKfzc/2 with map size 1073741824
14:38:12 [14:38:12.221]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:13 [14:38:13.295]
INFO:
prover-client:proving-broker Proving Broker started
14:38:13 [14:38:13.301]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1a769f8f8161d3fd settled=false
{"provingJobId":"1:BASE_PARITY:1a769f8f8161d3fd","status":"pending"}
14:38:13 [14:38:13.303]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6ff10eb3b832eff4 settled=false
{"provingJobId":"1:BASE_PARITY:6ff10eb3b832eff4","status":"pending"}
14:38:13 [14:38:13.304]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1a769f8f8161d3fd type=BASE_PARITY startedAt=1749825492245 older agent has taken job
{"provingJobId":"1:BASE_PARITY:1a769f8f8161d3fd"}
14:38:13 [14:38:13.305]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1a769f8f8161d3fd type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:1a769f8f8161d3fd"}
14:38:13 [14:38:13.308]
INFO:
prover-client:proving-broker Proving Broker started
14:38:13 [14:38:13.309]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ec6e41ca44bc6d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ec6e41ca44bc6d5"}
14:38:13 [14:38:13.310]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDvTA6y/1 with map size 1073741824
14:38:13 [14:38:13.313]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:13 [14:38:13.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9f8c1bcb246268f epochNumber=2
{"provingJobId":"1:BASE_PARITY:c9f8c1bcb246268f"}
14:38:13 [14:38:13.336]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDvTA6y/2 with map size 1073741824
14:38:13 [14:38:13.338]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:14 [14:38:14.357]
INFO:
prover-client:proving-broker Proving Broker started
14:38:14 [14:38:14.359]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4ec6e41ca44bc6d5 settled=false
{"provingJobId":"1:BASE_PARITY:4ec6e41ca44bc6d5","status":"pending"}
14:38:14 [14:38:14.360]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c9f8c1bcb246268f settled=false
{"provingJobId":"1:BASE_PARITY:c9f8c1bcb246268f","status":"pending"}
14:38:14 [14:38:14.361]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4ec6e41ca44bc6d5 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:4ec6e41ca44bc6d5"}
14:38:14 [14:38:14.364]
INFO:
prover-client:proving-broker Proving Broker started
14:38:14 [14:38:14.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40ea9231a9dcbd31 epochNumber=1
{"provingJobId":"1:BASE_PARITY:40ea9231a9dcbd31"}
14:38:14 [14:38:14.365]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsaOMXa/1 with map size 1073741824
14:38:14 [14:38:14.367]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:14 [14:38:14.386]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8239a5326122e4c epochNumber=2
{"provingJobId":"1:BASE_PARITY:e8239a5326122e4c"}
14:38:14 [14:38:14.386]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testsaOMXa/2 with map size 1073741824
14:38:14 [14:38:14.389]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:15 [14:38:15.405]
INFO:
prover-client:proving-broker Proving Broker started
14:38:15 [14:38:15.408]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:40ea9231a9dcbd31 settled=false
{"provingJobId":"1:BASE_PARITY:40ea9231a9dcbd31","status":"pending"}
14:38:15 [14:38:15.411]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e8239a5326122e4c settled=false
{"provingJobId":"1:BASE_PARITY:e8239a5326122e4c","status":"pending"}
14:38:15 [14:38:15.413]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:40ea9231a9dcbd31 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:40ea9231a9dcbd31"}
14:38:15 [14:38:15.414]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:40ea9231a9dcbd31 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:40ea9231a9dcbd31"}
14:38:15 [14:38:15.427]
INFO:
prover-client:proving-broker Proving Broker started
14:38:15 [14:38:15.428]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f372362378842c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f372362378842c6"}
14:38:15 [14:38:15.429]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0xlpD3/1 with map size 1073741824
14:38:15 [14:38:15.432]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:15 [14:38:15.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d9447a6d98c88a60 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d9447a6d98c88a60"}
14:38:15 [14:38:15.461]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0xlpD3/2 with map size 1073741824
14:38:15 [14:38:15.464]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:15 [14:38:15.486]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8f372362378842c6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8f372362378842c6"}
14:38:15 [14:38:15.494]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d9447a6d98c88a60 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d9447a6d98c88a60"}
14:38:15 [14:38:15.504]
INFO:
prover-client:proving-broker Proving Broker started
14:38:15 [14:38:15.505]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7aff6fc35b1bf834 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7aff6fc35b1bf834"}
14:38:15 [14:38:15.506]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5EHnrV/1 with map size 1073741824
14:38:15 [14:38:15.511]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:15 [14:38:15.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:afa0ee8835e0e87c epochNumber=2
{"provingJobId":"1:BASE_PARITY:afa0ee8835e0e87c"}
14:38:15 [14:38:15.535]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5EHnrV/2 with map size 1073741824
14:38:15 [14:38:15.538]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:15 [14:38:15.560]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7aff6fc35b1bf834 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7aff6fc35b1bf834"}
14:38:15 [14:38:15.560]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7aff6fc35b1bf834 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7aff6fc35b1bf834"}
14:38:15 [14:38:15.569]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:afa0ee8835e0e87c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:afa0ee8835e0e87c"}
14:38:15 [14:38:15.589]
INFO:
prover-client:proving-broker Proving Broker started
14:38:15 [14:38:15.590]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:56c08077098aa586 err=test error
{"provingJoId":"1:BASE_PARITY:56c08077098aa586"}
14:38:15 [14:38:15.601]
INFO:
prover-client:proving-broker Proving Broker started
14:38:15 [14:38:15.602]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1bfc8e3d4c6b52b5 not found
{"provingJobId":"1:BASE_PARITY:1bfc8e3d4c6b52b5"}
14:38:15 [14:38:15.613]
INFO:
prover-client:proving-broker Proving Broker started
14:38:15 [14:38:15.613]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6fe704abf57ec605 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6fe704abf57ec605"}
14:38:15 [14:38:15.614]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQSzEpa/1 with map size 1073741824
14:38:15 [14:38:15.620]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:15 [14:38:15.644]
INFO:
prover-client:proving-broker Proving Broker started
14:38:15 [14:38:15.644]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e6d510faa7cad1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e6d510faa7cad1f"}
14:38:15 [14:38:15.645]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsbPJi6/1 with map size 1073741824
14:38:15 [14:38:15.647]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:15 [14:38:15.781]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5e6d510faa7cad1f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5e6d510faa7cad1f"}
14:38:15 [14:38:15.795]
INFO:
prover-client:proving-broker Proving Broker started
14:38:15 [14:38:15.797]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5436f7d9fdda5a22 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5436f7d9fdda5a22"}
14:38:15 [14:38:15.798]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEAYib9/1 with map size 1073741824
14:38:15 [14:38:15.802]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:15 [14:38:15.947]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5436f7d9fdda5a22 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5436f7d9fdda5a22"}
14:38:15 [14:38:15.954]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60b15f3068462337 epochNumber=10
{"provingJobId":"1:BASE_PARITY:60b15f3068462337"}
14:38:15 [14:38:15.955]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testEAYib9/10 with map size 1073741824
14:38:15 [14:38:15.958]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:15 [14:38:15.998]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:15 [14:38:15.999]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:38:16 [14:38:16.109]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.110]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3dc3737f333e3675 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3dc3737f333e3675"}
14:38:16 [14:38:16.110]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvOWwqW/1 with map size 1073741824
14:38:16 [14:38:16.112]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.288]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3dc3737f333e3675 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:3dc3737f333e3675"}
14:38:16 [14:38:16.337]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.338]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:edb710b1f46a2398 epochNumber=1
{"provingJobId":"1:BASE_PARITY:edb710b1f46a2398"}
14:38:16 [14:38:16.339]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmO4maz/1 with map size 1073741824
14:38:16 [14:38:16.342]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.359]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:edb710b1f46a2398 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:edb710b1f46a2398"}
14:38:16 [14:38:16.364]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2585b2712ab00df epochNumber=1
{"provingJobId":"1:BASE_PARITY:c2585b2712ab00df"}
14:38:16 [14:38:16.368]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTdDf2T/1 with map size 1073741824
14:38:16 [14:38:16.370]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.390]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c2585b2712ab00df type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c2585b2712ab00df"}
14:38:16 [14:38:16.391]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c2585b2712ab00df type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:c2585b2712ab00df"}
14:38:16 [14:38:16.400]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.401]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98c30fd0f604d9f7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:98c30fd0f604d9f7"}
14:38:16 [14:38:16.403]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTEIxAv/1 with map size 1073741824
14:38:16 [14:38:16.405]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.427]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:98c30fd0f604d9f7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:98c30fd0f604d9f7"}
14:38:16 [14:38:16.436]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53450cc7453dacf8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53450cc7453dacf8"}
14:38:16 [14:38:16.438]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2Zr2Lx/1 with map size 1073741824
14:38:16 [14:38:16.440]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.460]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:53450cc7453dacf8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:53450cc7453dacf8"}
14:38:16 [14:38:16.462]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d10b47e2aa87681e epochNumber=3
{"provingJobId":"1:BASE_PARITY:d10b47e2aa87681e"}
14:38:16 [14:38:16.466]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test2Zr2Lx/3 with map size 1073741824
14:38:16 [14:38:16.471]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.512]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:16 [14:38:16.513]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:38:16 [14:38:16.518]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:53450cc7453dacf8 err=test error
{"provingJoId":"1:BASE_PARITY:53450cc7453dacf8"}
14:38:16 [14:38:16.524]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testN4cE7r/1 with map size 1073741824
14:38:16 [14:38:16.527]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.548]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testN4cE7r/2 with map size 1073741824
14:38:16 [14:38:16.552]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.573]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.576]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:622487aea69184eb settled=false
{"provingJobId":"1:BASE_PARITY:622487aea69184eb","status":"pending"}
14:38:16 [14:38:16.581]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5af33cea754c2c12 settled=false
{"provingJobId":"1:BASE_PARITY:5af33cea754c2c12","status":"pending"}
14:38:16 [14:38:16.588]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgAz4RI/1 with map size 1073741824
14:38:16 [14:38:16.592]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.611]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testgAz4RI/2 with map size 1073741824
14:38:16 [14:38:16.614]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.647]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.650]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fde6a073d439eb61 settled=true
{"provingJobId":"1:BASE_PARITY:fde6a073d439eb61","status":"fulfilled"}
14:38:16 [14:38:16.652]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:4403dd085fad5140 settled=true
{"provingJobId":"2:BASE_PARITY:4403dd085fad5140","status":"fulfilled"}
14:38:16 [14:38:16.665]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcSqKdw/1 with map size 1073741824
14:38:16 [14:38:16.667]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.700]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcSqKdw/2 with map size 1073741824
14:38:16 [14:38:16.701]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.719]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.721]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f934ec1752c1e96f settled=true
{"provingJobId":"1:BASE_PARITY:f934ec1752c1e96f","status":"fulfilled"}
14:38:16 [14:38:16.723]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a3129cbe61a817bc settled=false
{"provingJobId":"1:BASE_PARITY:a3129cbe61a817bc","status":"pending"}
14:38:16 [14:38:16.727]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.728]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53e94d102b0452b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53e94d102b0452b8"}
14:38:16 [14:38:16.728]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJZKnm5/1 with map size 1073741824
14:38:16 [14:38:16.731]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.752]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.753]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66c9124d80f03935 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66c9124d80f03935"}
14:38:16 [14:38:16.754]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:66c9124d80f03935: Error: db error: Error: db error
14:38:16 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:66c9124d80f03935"}
14:38:16 [14:38:16.758]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.758]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44095734942d9e55 epochNumber=1
{"provingJobId":"1:BASE_PARITY:44095734942d9e55"}
14:38:16 [14:38:16.759]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgXvmnE/1 with map size 1073741824
14:38:16 [14:38:16.763]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.785]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:44095734942d9e55 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:44095734942d9e55"}
14:38:16 [14:38:16.786]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:44095734942d9e55 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:44095734942d9e55"}
14:38:16 [14:38:16.797]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.798]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4f739552ab2470b epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4f739552ab2470b"}
14:38:16 [14:38:16.799]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWz2Ek9/1 with map size 1073741824
14:38:16 [14:38:16.801]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.819]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f4f739552ab2470b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f4f739552ab2470b"}
14:38:16 [14:38:16.819]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f4f739552ab2470b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f4f739552ab2470b"}
14:38:16 [14:38:16.820]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:f4f739552ab2470b: Error: db error
14:38:16 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:f4f739552ab2470b"}
14:38:16 [14:38:16.828]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.829]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf2d9994987ad52e epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf2d9994987ad52e"}
14:38:16 [14:38:16.830]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testixeogH/1 with map size 1073741824
14:38:16 [14:38:16.833]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.850]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bf2d9994987ad52e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bf2d9994987ad52e"}
14:38:16 [14:38:16.851]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bf2d9994987ad52e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bf2d9994987ad52e"}
14:38:16 [14:38:16.863]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.864]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0617da5f3ee6f1b2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0617da5f3ee6f1b2"}
14:38:16 [14:38:16.865]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testELE6c7/1 with map size 1073741824
14:38:16 [14:38:16.872]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.898]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0617da5f3ee6f1b2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0617da5f3ee6f1b2"}
14:38:16 [14:38:16.898]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0617da5f3ee6f1b2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0617da5f3ee6f1b2"}
14:38:16 [14:38:16.899]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:0617da5f3ee6f1b2 jobErr=test error: Error: db error
14:38:16 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:0617da5f3ee6f1b2"}
14:38:16 [14:38:16.903]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.904]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e4f971360fa6b9dc not found
{"provingJobId":"1:BASE_PARITY:e4f971360fa6b9dc"}
14:38:16 [14:38:16.914]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.915]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e615f1a1a63ab8ef err=test error
{"provingJoId":"1:BASE_PARITY:e615f1a1a63ab8ef"}
14:38:16 [14:38:16.919]
INFO:
prover-client:proving-broker Proving Broker started
14:38:16 [14:38:16.930]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:73e63a141647e342 epochNumber=1
{"provingJobId":"1:BASE_PARITY:73e63a141647e342"}
14:38:16 [14:38:16.931]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFEqHT0/1 with map size 1073741824
14:38:16 [14:38:16.933]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.953]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:73e63a141647e342 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:73e63a141647e342"}
14:38:16 [14:38:16.954]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:73e63a141647e342 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:73e63a141647e342"}
14:38:16 [14:38:16.970]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:c0cd4ba25c743591 epochNumber=2
{"provingJobId":"2:BASE_PARITY:c0cd4ba25c743591"}
14:38:16 [14:38:16.971]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFEqHT0/2 with map size 1073741824
14:38:16 [14:38:16.976]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:16 [14:38:16.995]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:c0cd4ba25c743591 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:c0cd4ba25c743591"}
14:38:16 [14:38:16.996]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c0cd4ba25c743591 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:c0cd4ba25c743591"}
14:38:17 [14:38:17.061]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:40fa000bf858436c epochNumber=3
{"provingJobId":"3:BASE_PARITY:40fa000bf858436c"}
14:38:17 [14:38:17.064]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testFEqHT0/3 with map size 1073741824
14:38:17 [14:38:17.070]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:17 [14:38:17.098]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:17 [14:38:17.099]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:38:17 [14:38:17.154]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:bff12fe73df1e145 epochNumber=4
{"provingJobId":"4:BASE_PARITY:bff12fe73df1e145"}
14:38:17 [14:38:17.155]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testFEqHT0/4 with map size 1073741824
14:38:17 [14:38:17.158]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:17 [14:38:17.206]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:17 [14:38:17.207]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
14:38:17 [14:38:17.254]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:cade40a9072c7958 epochNumber=5
{"provingJobId":"5:BASE_PARITY:cade40a9072c7958"}
14:38:17 [14:38:17.255]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testFEqHT0/5 with map size 1073741824
14:38:17 [14:38:17.260]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:38:17 [14:38:17.313]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:38:17 [14:38:17.314]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
14:38:17
PASS src/proving_broker/proving_broker.test.ts (
13.205 s)
14:38:17 ProvingBroker
14:38:17 Producer API
14:38:17
✓ refuses stale jobs (65 ms)
14:38:17
✓ enqueues jobs (3 ms)
14:38:17
✓ ignores duplicate jobs (4 ms)
14:38:17
✓ reports correct status when enqueuing repeat jobs (4 ms)
14:38:17
✓ throws an error in case of duplicate job IDs (27 ms)
14:38:17
✓ returns not-found status for non-existing jobs (2 ms)
14:38:17
✓ cancels jobs in queue (3 ms)
14:38:17
✓ cancels jobs in-progress (2 ms)
14:38:17
✓ returns job result if successful (3 ms)
14:38:17
✓ returns job error if failed (2 ms)
14:38:17
✓ correctly returns job status for concurrent writes (2 ms)
14:38:17
✓ refuses stale jobs (35 ms)
14:38:17
✓ enqueues jobs (35 ms)
14:38:17
✓ ignores duplicate jobs (28 ms)
14:38:17
✓ reports correct status when enqueuing repeat jobs (33 ms)
14:38:17
✓ throws an error in case of duplicate job IDs (28 ms)
14:38:17
✓ returns not-found status for non-existing jobs (3 ms)
14:38:17
✓ cancels jobs in queue (30 ms)
14:38:17
✓ cancels jobs in-progress (31 ms)
14:38:17
✓ returns job result if successful (29 ms)
14:38:17
✓ returns job error if failed (29 ms)
14:38:17
✓ correctly returns job status for concurrent writes (35 ms)
14:38:17 Consumer API
14:38:17
✓ returns undefined if no jobs are available (1 ms)
14:38:17
✓ returns jobs in priority order (1 ms)
14:38:17
✓ returns undefined if no jobs are available for the given allowList (1 ms)
14:38:17
✓ returns a job if it is in the allowList (2 ms)
14:38:17
✓ returns the most important job if it is in the allowList (2 ms)
14:38:17
✓ returns any job if filter is empty (16 ms)
14:38:17
✓ returns a new job when reporting job success (4 ms)
14:38:17
✓ returns a new job when reporting permanent error (6 ms)
14:38:17
✓ returns a new job when reporting retry-able error (3 ms)
14:38:17
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
14:38:17
✓ returns a new job if job is already in progress elsewhere (1058 ms)
14:38:17
✓ avoids sending the same job to a new agent after a restart (1004 ms)
14:38:17
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
14:38:17
✓ tracks job result if in progress (3 ms)
14:38:17
✓ tracks job result even if job is in queue (2 ms)
14:38:17
✓ ignores reported job error if unknown job (2 ms)
14:38:17
✓ ignores job result if unknown job (1 ms)
14:38:17
✓ returns undefined if no jobs are available (2 ms)
14:38:17
✓ returns jobs in priority order (44 ms)
14:38:17
✓ returns undefined if no jobs are available for the given allowList (20 ms)
14:38:17
✓ returns a job if it is in the allowList (75 ms)
14:38:17
✓ returns the most important job if it is in the allowList (63 ms)
14:38:17
✓ returns any job if filter is empty (65 ms)
14:38:17
✓ returns a new job when reporting job success (34 ms)
14:38:17
✓ returns a new job when reporting permanent error (39 ms)
14:38:17
✓ returns a new job when reporting retry-able error (35 ms)
14:38:17
✓ returns a new job when reporting progress if current one is cancelled (45 ms)
14:38:17
✓ returns a new job if job is already in progress elsewhere (1125 ms)
14:38:17
✓ avoids sending the same job to a new agent after a restart (1055 ms)
14:38:17
✓ avoids sending a completed job to a new agent after a restart (1060 ms)
14:38:17
✓ tracks job result if in progress (78 ms)
14:38:17
✓ tracks job result even if job is in queue (80 ms)
14:38:17
✓ ignores reported job error if unknown job (11 ms)
14:38:17
✓ ignores job result if unknown job (11 ms)
14:38:17 Timeouts
14:38:17
✓ tracks in progress jobs (3 ms)
14:38:17
✓ re-enqueues jobs that time out (110 ms)
14:38:17
✓ cancel stale jobs that time out (229 ms)
14:38:17
✓ keeps the jobs in progress while it is alive (204 ms)
14:38:17
✓ tracks in progress jobs (37 ms)
14:38:17
✓ re-enqueues jobs that time out (145 ms)
14:38:17
✓ cancel stale jobs that time out (317 ms)
14:38:17
✓ keeps the jobs in progress while it is alive (227 ms)
14:38:17 Retries
14:38:17
✓ retries jobs (2 ms)
14:38:17
✓ retries up to a maximum number of times (2 ms)
14:38:17
✓ passing retry=false does not retry (2 ms)
14:38:17
✓ does not retry if job is stale (28 ms)
14:38:17
✓ retries jobs (26 ms)
14:38:17
✓ retries up to a maximum number of times (37 ms)
14:38:17
✓ passing retry=false does not retry (36 ms)
14:38:17
✓ does not retry if job is stale (85 ms)
14:38:17 Database management
14:38:17
✓ re-enqueues proof requests on start (2 ms)
14:38:17
✓ restores proof results on start (2 ms)
14:38:17
✓ only re-enqueues unfinished jobs (2 ms)
14:38:17
✓ saves job when enqueued (2 ms)
14:38:17
✓ does not retain job if database fails to save (2 ms)
14:38:17
✓ saves job result (1 ms)
14:38:17
✓ saves result even if database fails to save (2 ms)
14:38:17
✓ saves job error (5 ms)
14:38:17
✓ saves job error even if database fails to save (3 ms)
14:38:17
✓ does not save job result if job is unknown (2 ms)
14:38:17
✓ does not save job error if job is unknown (1 ms)
14:38:17
✓ cleans up old jobs periodically (259 ms)
14:38:17
✓ re-enqueues proof requests on start (64 ms)
14:38:17
✓ restores proof results on start (71 ms)
14:38:17
✓ only re-enqueues unfinished jobs (69 ms)
14:38:17
✓ saves job when enqueued (23 ms)
14:38:17
✓ does not retain job if database fails to save (6 ms)
14:38:17
✓ saves job result (38 ms)
14:38:17
✓ saves result even if database fails to save (27 ms)
14:38:17
✓ saves job error (37 ms)
14:38:17
✓ saves job error even if database fails to save (39 ms)
14:38:17
✓ does not save job result if job is unknown (6 ms)
14:38:17
✓ does not save job error if job is unknown (9 ms)
14:38:17
✓ cleans up old jobs periodically (424 ms)
14:38:17
14:38:17
Test Suites: 1 passed, 1 total
14:38:17
Tests: 96 passed, 96 total
14:38:17
Snapshots: 0 total
14:38:17
Time: 13.297 s
14:38:17
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
14:38:17
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?