Parent Log:
http://ci.aztec-labs.com/89e702b6b9f7cdc0
Command: 5b7486e09cfa5726 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/d5c48d855d83e19712c6b72281801c6fce176ec1
Env: REF_NAME=gh-readonly-queue/next/pr-15128-8cc8a56892d36191eefb49254ebbe49ea99ef4b8 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Wed Jun 18 11:38:41 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15128_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
11:38:46 [11:38:46.076]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.081]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2edd7fbae6240101 epochNumber=42
{"provingJobId":"1:BASE_PARITY:2edd7fbae6240101"}
11:38:46 [11:38:46.084]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:d4d9620e1a09e06a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4d9620e1a09e06a"}
11:38:46 [11:38:46.109]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.110]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8497294d86d9f613 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8497294d86d9f613"}
11:38:46 [11:38:46.112]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:237fc7df9b7c01d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:237fc7df9b7c01d4"}
11:38:46 [11:38:46.114]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.115]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2dad097990ef66ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:2dad097990ef66ed"}
11:38:46 [11:38:46.116]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2dad097990ef66ed epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2dad097990ef66ed"}
11:38:46 [11:38:46.119]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a98db5718b58ab0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a98db5718b58ab0"}
11:38:46 [11:38:46.121]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1a98db5718b58ab0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1a98db5718b58ab0"}
11:38:46 [11:38:46.121]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1a98db5718b58ab0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1a98db5718b58ab0"}
11:38:46 [11:38:46.122]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1a98db5718b58ab0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1a98db5718b58ab0"}
11:38:46 [11:38:46.124]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.125]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34e0f500eacf05e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:34e0f500eacf05e1"}
11:38:46 [11:38:46.146]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.147]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.148]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f077e9efc05ea78 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f077e9efc05ea78"}
11:38:46 [11:38:46.149]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0f077e9efc05ea78
{"provingJobId":"1:BASE_PARITY:0f077e9efc05ea78"}
11:38:46 [11:38:46.149]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0f077e9efc05ea78 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0f077e9efc05ea78"}
11:38:46 [11:38:46.149]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0f077e9efc05ea78 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:0f077e9efc05ea78"}
11:38:46 [11:38:46.151]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.151]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a0517921c22449e epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a0517921c22449e"}
11:38:46 [11:38:46.152]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1a0517921c22449e
{"provingJobId":"1:BASE_PARITY:1a0517921c22449e"}
11:38:46 [11:38:46.152]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1a0517921c22449e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1a0517921c22449e"}
11:38:46 [11:38:46.153]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.154]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd6c635bf8860483 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd6c635bf8860483"}
11:38:46 [11:38:46.154]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd6c635bf8860483 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fd6c635bf8860483"}
11:38:46 [11:38:46.154]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fd6c635bf8860483 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fd6c635bf8860483"}
11:38:46 [11:38:46.155]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.156]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46bbfc5a65da2248 epochNumber=1
{"provingJobId":"1:BASE_PARITY:46bbfc5a65da2248"}
11:38:46 [11:38:46.156]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:46bbfc5a65da2248 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:46bbfc5a65da2248"}
11:38:46 [11:38:46.156]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:46bbfc5a65da2248 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:46bbfc5a65da2248"}
11:38:46 [11:38:46.157]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.157]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95034a925813401f epochNumber=0
{"provingJobId":"1:BASE_PARITY:95034a925813401f"}
11:38:46 [11:38:46.157]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:95034a925813401f epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:95034a925813401f"}
11:38:46 [11:38:46.158]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0766a343e628d6be epochNumber=0
{"provingJobId":"1:BASE_PARITY:0766a343e628d6be"}
11:38:46 [11:38:46.158]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:95034a925813401f epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:95034a925813401f"}
11:38:46 [11:38:46.158]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e59bde6f0be55e0a epochNumber=0
{"provingJobId":"1:BASE_PARITY:e59bde6f0be55e0a"}
11:38:46 [11:38:46.166]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.168]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:771bf44260284353 epochNumber=2
{"provingJobId":"1:BASE_PARITY:771bf44260284353"}
11:38:46 [11:38:46.169]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b34d5dd6d025d0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:5b34d5dd6d025d0e"}
11:38:46 [11:38:46.170]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.170]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16cb7553819d4207 epochNumber=1
{"provingJobId":"1:BASE_PARITY:16cb7553819d4207"}
11:38:46 [11:38:46.171]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.171]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89a6c435046dee5f epochNumber=1
{"provingJobId":"1:BASE_PARITY:89a6c435046dee5f"}
11:38:46 [11:38:46.171]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83a72d4953bf4ba6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:83a72d4953bf4ba6"}
11:38:46 [11:38:46.172]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42fc581b83151bfb epochNumber=2
{"provingJobId":"1:BASE_PARITY:42fc581b83151bfb"}
11:38:46 [11:38:46.172]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e3a5915c6e5f133 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e3a5915c6e5f133"}
11:38:46 [11:38:46.173]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.173]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:674cf24f825d4509 epochNumber=1
{"provingJobId":"1:BASE_PARITY:674cf24f825d4509"}
11:38:46 [11:38:46.173]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83218d05899b734e epochNumber=1
{"provingJobId":"1:BASE_PARITY:83218d05899b734e"}
11:38:46 [11:38:46.174]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d0b70d3abd7c9ac6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d0b70d3abd7c9ac6"}
11:38:46 [11:38:46.174]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4ada8d200eece1d epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4ada8d200eece1d"}
11:38:46 [11:38:46.175]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.175]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ea171e09a7afd12 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1ea171e09a7afd12"}
11:38:46 [11:38:46.176]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da273ead342b1103 epochNumber=1
{"provingJobId":"1:BASE_PARITY:da273ead342b1103"}
11:38:46 [11:38:46.176]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9dd30052d023c55e epochNumber=2
{"provingJobId":"1:BASE_PARITY:9dd30052d023c55e"}
11:38:46 [11:38:46.176]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f772ee522a8c08e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f772ee522a8c08e7"}
11:38:46 [11:38:46.177]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.177]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2dffa0ce109f2f38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2dffa0ce109f2f38"}
11:38:46 [11:38:46.178]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:828e2e66b33794c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:828e2e66b33794c8"}
11:38:46 [11:38:46.178]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2dffa0ce109f2f38 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2dffa0ce109f2f38"}
11:38:46 [11:38:46.179]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.179]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3555855ee8bff13 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3555855ee8bff13"}
11:38:46 [11:38:46.180]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ff889570f0ab549 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ff889570f0ab549"}
11:38:46 [11:38:46.180]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d3555855ee8bff13 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:d3555855ee8bff13"}
11:38:46 [11:38:46.181]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.181]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80201c194118ff0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:80201c194118ff0e"}
11:38:46 [11:38:46.181]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f003a0789f19fd11 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f003a0789f19fd11"}
11:38:46 [11:38:46.182]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:80201c194118ff0e type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:80201c194118ff0e"}
11:38:46 [11:38:46.182]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.183]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d1b4a2c798b9590 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5d1b4a2c798b9590"}
11:38:46 [11:38:46.183]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5d1b4a2c798b9590
{"provingJobId":"1:BASE_PARITY:5d1b4a2c798b9590"}
11:38:46 [11:38:46.183]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5d1b4a2c798b9590 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5d1b4a2c798b9590"}
11:38:46 [11:38:46.184]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f30956fc144ff13 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f30956fc144ff13"}
11:38:46 [11:38:46.184]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5d1b4a2c798b9590 has already been completed
{"provingJobId":"1:BASE_PARITY:5d1b4a2c798b9590"}
11:38:46 [11:38:46.185]
INFO:
prover-client:proving-broker Proving Broker started
11:38:46 [11:38:46.185]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26fe05ffb6727086 epochNumber=1
{"provingJobId":"1:BASE_PARITY:26fe05ffb6727086"}
11:38:46 [11:38:46.186]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f10049be7e8b7f4a epochNumber=2
{"provingJobId":"1:BASE_PARITY:f10049be7e8b7f4a"}
11:38:47 [11:38:47.238]
INFO:
prover-client:proving-broker Proving Broker started
11:38:47 [11:38:47.239]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:26fe05ffb6727086 settled=false
{"provingJobId":"1:BASE_PARITY:26fe05ffb6727086","status":"pending"}
11:38:47 [11:38:47.239]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f10049be7e8b7f4a settled=false
{"provingJobId":"1:BASE_PARITY:f10049be7e8b7f4a","status":"pending"}
11:38:47 [11:38:47.240]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:26fe05ffb6727086 type=BASE_PARITY startedAt=1750246726186 older agent has taken job
{"provingJobId":"1:BASE_PARITY:26fe05ffb6727086"}
11:38:47 [11:38:47.241]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:26fe05ffb6727086 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:26fe05ffb6727086"}
11:38:47 [11:38:47.242]
INFO:
prover-client:proving-broker Proving Broker started
11:38:47 [11:38:47.243]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f59ac529f9fe757d epochNumber=1
{"provingJobId":"1:BASE_PARITY:f59ac529f9fe757d"}
11:38:47 [11:38:47.243]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c7339a388ad23138 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c7339a388ad23138"}
11:38:48 [11:38:48.244]
INFO:
prover-client:proving-broker Proving Broker started
11:38:48 [11:38:48.245]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f59ac529f9fe757d settled=false
{"provingJobId":"1:BASE_PARITY:f59ac529f9fe757d","status":"pending"}
11:38:48 [11:38:48.245]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c7339a388ad23138 settled=false
{"provingJobId":"1:BASE_PARITY:c7339a388ad23138","status":"pending"}
11:38:48 [11:38:48.246]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f59ac529f9fe757d type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:f59ac529f9fe757d"}
11:38:48 [11:38:48.247]
INFO:
prover-client:proving-broker Proving Broker started
11:38:48 [11:38:48.248]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60553e352ba6987b epochNumber=1
{"provingJobId":"1:BASE_PARITY:60553e352ba6987b"}
11:38:48 [11:38:48.248]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03f7f65b0414e617 epochNumber=2
{"provingJobId":"1:BASE_PARITY:03f7f65b0414e617"}
11:38:49 [11:38:49.248]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.249]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:60553e352ba6987b settled=false
{"provingJobId":"1:BASE_PARITY:60553e352ba6987b","status":"pending"}
11:38:49 [11:38:49.249]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:03f7f65b0414e617 settled=false
{"provingJobId":"1:BASE_PARITY:03f7f65b0414e617","status":"pending"}
11:38:49 [11:38:49.250]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:60553e352ba6987b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:60553e352ba6987b"}
11:38:49 [11:38:49.250]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:60553e352ba6987b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:60553e352ba6987b"}
11:38:49 [11:38:49.251]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.252]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43c6b24806d1e8d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43c6b24806d1e8d6"}
11:38:49 [11:38:49.252]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:146fade826a45c30 epochNumber=2
{"provingJobId":"1:BASE_PARITY:146fade826a45c30"}
11:38:49 [11:38:49.253]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:43c6b24806d1e8d6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:43c6b24806d1e8d6"}
11:38:49 [11:38:49.253]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:146fade826a45c30 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:146fade826a45c30"}
11:38:49 [11:38:49.254]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75cf161b2199de5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:75cf161b2199de5d"}
11:38:49 [11:38:49.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60b8d6a6bcd54c9a epochNumber=2
{"provingJobId":"1:BASE_PARITY:60b8d6a6bcd54c9a"}
11:38:49 [11:38:49.255]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:75cf161b2199de5d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:75cf161b2199de5d"}
11:38:49 [11:38:49.255]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:75cf161b2199de5d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:75cf161b2199de5d"}
11:38:49 [11:38:49.256]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:60b8d6a6bcd54c9a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:60b8d6a6bcd54c9a"}
11:38:49 [11:38:49.256]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.257]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6f6ebb3063b894ca err=test error
{"provingJoId":"1:BASE_PARITY:6f6ebb3063b894ca"}
11:38:49 [11:38:49.257]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.258]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1440694eaca3370a not found
{"provingJobId":"1:BASE_PARITY:1440694eaca3370a"}
11:38:49 [11:38:49.259]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.259]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:37c1db4fa903b966 epochNumber=1
{"provingJobId":"1:BASE_PARITY:37c1db4fa903b966"}
11:38:49 [11:38:49.260]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:628ca1943fe1c4de epochNumber=1
{"provingJobId":"1:BASE_PARITY:628ca1943fe1c4de"}
11:38:49 [11:38:49.363]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:628ca1943fe1c4de timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:628ca1943fe1c4de"}
11:38:49 [11:38:49.370]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.371]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4551f18d24992e51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4551f18d24992e51"}
11:38:49 [11:38:49.472]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4551f18d24992e51 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4551f18d24992e51"}
11:38:49 [11:38:49.474]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6e993a1525e1314 epochNumber=10
{"provingJobId":"1:BASE_PARITY:a6e993a1525e1314"}
11:38:49 [11:38:49.499]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:49 [11:38:49.601]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.602]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ed5a62c0476384c epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ed5a62c0476384c"}
11:38:49 [11:38:49.778]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4ed5a62c0476384c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4ed5a62c0476384c"}
11:38:49 [11:38:49.805]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fdee19217d371d95 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fdee19217d371d95"}
11:38:49 [11:38:49.807]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fdee19217d371d95 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:fdee19217d371d95"}
11:38:49 [11:38:49.808]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.809]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:096f73412b25105a epochNumber=1
{"provingJobId":"1:BASE_PARITY:096f73412b25105a"}
11:38:49 [11:38:49.809]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:096f73412b25105a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:096f73412b25105a"}
11:38:49 [11:38:49.810]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:096f73412b25105a type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:096f73412b25105a"}
11:38:49 [11:38:49.811]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.812]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:856ee2236cefe9bc epochNumber=1
{"provingJobId":"1:BASE_PARITY:856ee2236cefe9bc"}
11:38:49 [11:38:49.812]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:856ee2236cefe9bc type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:856ee2236cefe9bc"}
11:38:49 [11:38:49.814]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.815]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b511bacd4f591644 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b511bacd4f591644"}
11:38:49 [11:38:49.816]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b511bacd4f591644 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b511bacd4f591644"}
11:38:49 [11:38:49.816]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0afdc4c55b38aeed epochNumber=3
{"provingJobId":"1:BASE_PARITY:0afdc4c55b38aeed"}
11:38:49 [11:38:49.841]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:49 [11:38:49.842]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b511bacd4f591644 err=test error
{"provingJoId":"1:BASE_PARITY:b511bacd4f591644"}
11:38:49 [11:38:49.844]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.845]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:baf87b4a707a62ad settled=false
{"provingJobId":"1:BASE_PARITY:baf87b4a707a62ad","status":"pending"}
11:38:49 [11:38:49.845]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:227d90b8e23c898b settled=false
{"provingJobId":"1:BASE_PARITY:227d90b8e23c898b","status":"pending"}
11:38:49 [11:38:49.848]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.849]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:48c27c79eec6d0ce settled=true
{"provingJobId":"1:BASE_PARITY:48c27c79eec6d0ce","status":"fulfilled"}
11:38:49 [11:38:49.849]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:1f6bc803c51c89b8 settled=true
{"provingJobId":"2:BASE_PARITY:1f6bc803c51c89b8","status":"fulfilled"}
11:38:49 [11:38:49.851]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.851]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0eae5cc01cace68a settled=true
{"provingJobId":"1:BASE_PARITY:0eae5cc01cace68a","status":"fulfilled"}
11:38:49 [11:38:49.851]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ba6daecce3f9559a settled=false
{"provingJobId":"1:BASE_PARITY:ba6daecce3f9559a","status":"pending"}
11:38:49 [11:38:49.852]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.854]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c603934928bcf6ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:c603934928bcf6ba"}
11:38:49 [11:38:49.855]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.855]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:18085349106b0858 epochNumber=1
{"provingJobId":"1:BASE_PARITY:18085349106b0858"}
11:38:49 [11:38:49.856]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:18085349106b0858: Error: db error: Error: db error
11:38:49 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:18085349106b0858"}
11:38:49 [11:38:49.857]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.857]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5bafe7027056ce7a epochNumber=1
{"provingJobId":"1:BASE_PARITY:5bafe7027056ce7a"}
11:38:49 [11:38:49.857]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5bafe7027056ce7a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5bafe7027056ce7a"}
11:38:49 [11:38:49.858]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5bafe7027056ce7a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5bafe7027056ce7a"}
11:38:49 [11:38:49.858]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.859]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c456ee29c47a696c epochNumber=1
{"provingJobId":"1:BASE_PARITY:c456ee29c47a696c"}
11:38:49 [11:38:49.859]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c456ee29c47a696c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c456ee29c47a696c"}
11:38:49 [11:38:49.859]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c456ee29c47a696c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c456ee29c47a696c"}
11:38:49 [11:38:49.860]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:c456ee29c47a696c: Error: db error
11:38:49 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:c456ee29c47a696c"}
11:38:49 [11:38:49.860]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.861]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd8897b3c9f16274 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd8897b3c9f16274"}
11:38:49 [11:38:49.861]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dd8897b3c9f16274 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dd8897b3c9f16274"}
11:38:49 [11:38:49.861]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dd8897b3c9f16274 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:dd8897b3c9f16274"}
11:38:49 [11:38:49.862]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eba640456642c440 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eba640456642c440"}
11:38:49 [11:38:49.862]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eba640456642c440 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:eba640456642c440"}
11:38:49 [11:38:49.863]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:eba640456642c440 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:eba640456642c440"}
11:38:49 [11:38:49.863]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:eba640456642c440 jobErr=test error: Error: db error
11:38:49 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:eba640456642c440"}
11:38:49 [11:38:49.864]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.865]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b7f582c6d7f1b107 not found
{"provingJobId":"1:BASE_PARITY:b7f582c6d7f1b107"}
11:38:49 [11:38:49.867]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.867]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:656c6e2d2397db59 err=test error
{"provingJoId":"1:BASE_PARITY:656c6e2d2397db59"}
11:38:49 [11:38:49.868]
INFO:
prover-client:proving-broker Proving Broker started
11:38:49 [11:38:49.878]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bcafe12878798cca epochNumber=1
{"provingJobId":"1:BASE_PARITY:bcafe12878798cca"}
11:38:49 [11:38:49.878]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bcafe12878798cca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bcafe12878798cca"}
11:38:49 [11:38:49.879]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bcafe12878798cca type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bcafe12878798cca"}
11:38:49 [11:38:49.890]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:d603801e6d669796 epochNumber=2
{"provingJobId":"2:BASE_PARITY:d603801e6d669796"}
11:38:49 [11:38:49.890]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:d603801e6d669796 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:d603801e6d669796"}
11:38:49 [11:38:49.890]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:d603801e6d669796 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:d603801e6d669796"}
11:38:49 [11:38:49.951]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:356e7a3f1d99582a epochNumber=3
{"provingJobId":"3:BASE_PARITY:356e7a3f1d99582a"}
11:38:49 [11:38:49.970]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:50 [11:38:50.013]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:8ce824b12403990d epochNumber=4
{"provingJobId":"4:BASE_PARITY:8ce824b12403990d"}
11:38:50 [11:38:50.020]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:50 [11:38:50.079]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:153b5790e8d2541f epochNumber=5
{"provingJobId":"5:BASE_PARITY:153b5790e8d2541f"}
11:38:50 [11:38:50.097]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:50 [11:38:50.136]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.136]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2cd729e6e5560295 epochNumber=42
{"provingJobId":"1:BASE_PARITY:2cd729e6e5560295"}
11:38:50 [11:38:50.137]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testcW3j08/42 with map size 1073741824
11:38:50 [11:38:50.142]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.165]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:49798ae133477d22 epochNumber=1
{"provingJobId":"1:BASE_PARITY:49798ae133477d22"}
11:38:50 [11:38:50.169]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.170]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b116e5f46958c5c epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b116e5f46958c5c"}
11:38:50 [11:38:50.171]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7PA2x7/1 with map size 1073741824
11:38:50 [11:38:50.172]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.191]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb6800138a56acae epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb6800138a56acae"}
11:38:50 [11:38:50.199]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81e92d21b3832842 epochNumber=1
{"provingJobId":"1:BASE_PARITY:81e92d21b3832842"}
11:38:50 [11:38:50.201]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnVWh79/1 with map size 1073741824
11:38:50 [11:38:50.203]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.227]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:81e92d21b3832842 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:81e92d21b3832842"}
11:38:50 [11:38:50.232]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3e16a6d04d85d88 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3e16a6d04d85d88"}
11:38:50 [11:38:50.233]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testte00Hv/1 with map size 1073741824
11:38:50 [11:38:50.235]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.254]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a3e16a6d04d85d88 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:a3e16a6d04d85d88"}
11:38:50 [11:38:50.254]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a3e16a6d04d85d88 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a3e16a6d04d85d88"}
11:38:50 [11:38:50.262]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a3e16a6d04d85d88 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:a3e16a6d04d85d88"}
11:38:50 [11:38:50.265]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:574118fcc3173260 epochNumber=1
{"provingJobId":"1:BASE_PARITY:574118fcc3173260"}
11:38:50 [11:38:50.266]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmtrtzn/1 with map size 1073741824
11:38:50 [11:38:50.270]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.294]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.298]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.299]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0e22c54757d1683 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0e22c54757d1683"}
11:38:50 [11:38:50.300]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxB0XX2/1 with map size 1073741824
11:38:50 [11:38:50.302]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.330]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e0e22c54757d1683
{"provingJobId":"1:BASE_PARITY:e0e22c54757d1683"}
11:38:50 [11:38:50.331]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e0e22c54757d1683 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e0e22c54757d1683"}
11:38:50 [11:38:50.331]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e0e22c54757d1683 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:e0e22c54757d1683"}
11:38:50 [11:38:50.339]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.340]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3aad93fc4b2b7d84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3aad93fc4b2b7d84"}
11:38:50 [11:38:50.340]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testctBD9x/1 with map size 1073741824
11:38:50 [11:38:50.342]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.358]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3aad93fc4b2b7d84
{"provingJobId":"1:BASE_PARITY:3aad93fc4b2b7d84"}
11:38:50 [11:38:50.358]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3aad93fc4b2b7d84 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3aad93fc4b2b7d84"}
11:38:50 [11:38:50.369]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.369]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a397335d126d0715 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a397335d126d0715"}
11:38:50 [11:38:50.370]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGcEW8f/1 with map size 1073741824
11:38:50 [11:38:50.375]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.397]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a397335d126d0715 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a397335d126d0715"}
11:38:50 [11:38:50.398]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a397335d126d0715 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a397335d126d0715"}
11:38:50 [11:38:50.416]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82e69fffa3fc269d epochNumber=1
{"provingJobId":"1:BASE_PARITY:82e69fffa3fc269d"}
11:38:50 [11:38:50.418]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGP8djl/1 with map size 1073741824
11:38:50 [11:38:50.425]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.446]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:82e69fffa3fc269d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:82e69fffa3fc269d"}
11:38:50 [11:38:50.447]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:82e69fffa3fc269d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:82e69fffa3fc269d"}
11:38:50 [11:38:50.457]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.458]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:448d2672b550da80 epochNumber=0
{"provingJobId":"1:BASE_PARITY:448d2672b550da80"}
11:38:50 [11:38:50.460]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testR7RW6s/0 with map size 1073741824
11:38:50 [11:38:50.462]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.482]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:448d2672b550da80 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:448d2672b550da80"}
11:38:50 [11:38:50.483]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75cc5cdada03f3d6 epochNumber=0
{"provingJobId":"1:BASE_PARITY:75cc5cdada03f3d6"}
11:38:50 [11:38:50.483]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:448d2672b550da80 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:448d2672b550da80"}
11:38:50 [11:38:50.483]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c422d8f0a8f6044f epochNumber=0
{"provingJobId":"1:BASE_PARITY:c422d8f0a8f6044f"}
11:38:50 [11:38:50.497]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.500]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5be60399c3cf8f57 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5be60399c3cf8f57"}
11:38:50 [11:38:50.500]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testhknYXG/2 with map size 1073741824
11:38:50 [11:38:50.502]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ad628f95220407f epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ad628f95220407f"}
11:38:50 [11:38:50.541]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhknYXG/1 with map size 1073741824
11:38:50 [11:38:50.543]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.560]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.560]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64e1973765295833 epochNumber=1
{"provingJobId":"1:BASE_PARITY:64e1973765295833"}
11:38:50 [11:38:50.561]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlWBsPM/1 with map size 1073741824
11:38:50 [11:38:50.563]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.581]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8000bebc4d23e864 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8000bebc4d23e864"}
11:38:50 [11:38:50.582]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhqkjJn/1 with map size 1073741824
11:38:50 [11:38:50.584]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.601]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d81cdcb1414cb9dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:d81cdcb1414cb9dd"}
11:38:50 [11:38:50.610]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:257b18ebebe46a7a epochNumber=2
{"provingJobId":"1:BASE_PARITY:257b18ebebe46a7a"}
11:38:50 [11:38:50.616]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testhqkjJn/2 with map size 1073741824
11:38:50 [11:38:50.622]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.641]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf49a80384e4f8f4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf49a80384e4f8f4"}
11:38:50 [11:38:50.652]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.652]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06f307fedb8ef368 epochNumber=1
{"provingJobId":"1:BASE_PARITY:06f307fedb8ef368"}
11:38:50 [11:38:50.653]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2XNIle/1 with map size 1073741824
11:38:50 [11:38:50.655]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.673]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7467ba920961726a epochNumber=1
{"provingJobId":"1:BASE_PARITY:7467ba920961726a"}
11:38:50 [11:38:50.685]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ceb15d1c3f33dda epochNumber=2
{"provingJobId":"1:BASE_PARITY:2ceb15d1c3f33dda"}
11:38:50 [11:38:50.686]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2XNIle/2 with map size 1073741824
11:38:50 [11:38:50.689]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.719]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1efcce795dd7c249 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1efcce795dd7c249"}
11:38:50 [11:38:50.733]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.734]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54d348e08673ae0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:54d348e08673ae0f"}
11:38:50 [11:38:50.735]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6l1rj0/1 with map size 1073741824
11:38:50 [11:38:50.738]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6bff59783e450a8c epochNumber=1
{"provingJobId":"1:BASE_PARITY:6bff59783e450a8c"}
11:38:50 [11:38:50.770]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8cc160f898cd66dc epochNumber=2
{"provingJobId":"1:BASE_PARITY:8cc160f898cd66dc"}
11:38:50 [11:38:50.771]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6l1rj0/2 with map size 1073741824
11:38:50 [11:38:50.774]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f83e150628f6afd epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f83e150628f6afd"}
11:38:50 [11:38:50.805]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf363561e372fc35 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf363561e372fc35"}
11:38:50 [11:38:50.808]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCo5v50/1 with map size 1073741824
11:38:50 [11:38:50.812]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.830]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f96b2ee58e870e6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f96b2ee58e870e6c"}
11:38:50 [11:38:50.837]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cf363561e372fc35 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cf363561e372fc35"}
11:38:50 [11:38:50.853]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.854]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82153b034b938930 epochNumber=1
{"provingJobId":"1:BASE_PARITY:82153b034b938930"}
11:38:50 [11:38:50.854]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPLgtLk/1 with map size 1073741824
11:38:50 [11:38:50.856]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.878]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61c255e46cdff628 epochNumber=1
{"provingJobId":"1:BASE_PARITY:61c255e46cdff628"}
11:38:50 [11:38:50.887]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:82153b034b938930 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:82153b034b938930"}
11:38:50 [11:38:50.897]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.897]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b814a528d6adc6f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b814a528d6adc6f"}
11:38:50 [11:38:50.899]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testh0pJsc/1 with map size 1073741824
11:38:50 [11:38:50.901]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.928]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:37a999bccca91c8b epochNumber=1
{"provingJobId":"1:BASE_PARITY:37a999bccca91c8b"}
11:38:50 [11:38:50.935]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4b814a528d6adc6f type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:4b814a528d6adc6f"}
11:38:50 [11:38:50.938]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05aa3db1b4aa9733 epochNumber=1
{"provingJobId":"1:BASE_PARITY:05aa3db1b4aa9733"}
11:38:50 [11:38:50.940]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt9yHF3/1 with map size 1073741824
11:38:50 [11:38:50.942]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:50 [11:38:50.963]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:05aa3db1b4aa9733
{"provingJobId":"1:BASE_PARITY:05aa3db1b4aa9733"}
11:38:50 [11:38:50.964]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:05aa3db1b4aa9733 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:05aa3db1b4aa9733"}
11:38:50 [11:38:50.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:067888208484d5af epochNumber=1
{"provingJobId":"1:BASE_PARITY:067888208484d5af"}
11:38:50 [11:38:50.976]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:05aa3db1b4aa9733 has already been completed
{"provingJobId":"1:BASE_PARITY:05aa3db1b4aa9733"}
11:38:50 [11:38:50.979]
INFO:
prover-client:proving-broker Proving Broker started
11:38:50 [11:38:50.980]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:860aa392c65b918e epochNumber=1
{"provingJobId":"1:BASE_PARITY:860aa392c65b918e"}
11:38:50 [11:38:50.980]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfylcyP/1 with map size 1073741824
11:38:50 [11:38:50.984]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:51 [11:38:51.007]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc15b4b3ef29f739 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cc15b4b3ef29f739"}
11:38:51 [11:38:51.008]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfylcyP/2 with map size 1073741824
11:38:51 [11:38:51.013]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:52 [11:38:52.083]
INFO:
prover-client:proving-broker Proving Broker started
11:38:52 [11:38:52.087]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:860aa392c65b918e settled=false
{"provingJobId":"1:BASE_PARITY:860aa392c65b918e","status":"pending"}
11:38:52 [11:38:52.089]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cc15b4b3ef29f739 settled=false
{"provingJobId":"1:BASE_PARITY:cc15b4b3ef29f739","status":"pending"}
11:38:52 [11:38:52.090]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:860aa392c65b918e type=BASE_PARITY startedAt=1750246731031 older agent has taken job
{"provingJobId":"1:BASE_PARITY:860aa392c65b918e"}
11:38:52 [11:38:52.091]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:860aa392c65b918e type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:860aa392c65b918e"}
11:38:52 [11:38:52.095]
INFO:
prover-client:proving-broker Proving Broker started
11:38:52 [11:38:52.096]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72180f41ada162a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:72180f41ada162a1"}
11:38:52 [11:38:52.097]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5ZVXvP/1 with map size 1073741824
11:38:52 [11:38:52.099]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:52 [11:38:52.120]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b268f77d0eeae76a epochNumber=2
{"provingJobId":"1:BASE_PARITY:b268f77d0eeae76a"}
11:38:52 [11:38:52.122]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5ZVXvP/2 with map size 1073741824
11:38:52 [11:38:52.126]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:53 [11:38:53.147]
INFO:
prover-client:proving-broker Proving Broker started
11:38:53 [11:38:53.149]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:72180f41ada162a1 settled=false
{"provingJobId":"1:BASE_PARITY:72180f41ada162a1","status":"pending"}
11:38:53 [11:38:53.151]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b268f77d0eeae76a settled=false
{"provingJobId":"1:BASE_PARITY:b268f77d0eeae76a","status":"pending"}
11:38:53 [11:38:53.151]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:72180f41ada162a1 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:72180f41ada162a1"}
11:38:53 [11:38:53.155]
INFO:
prover-client:proving-broker Proving Broker started
11:38:53 [11:38:53.155]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bae1de3d178fe40e epochNumber=1
{"provingJobId":"1:BASE_PARITY:bae1de3d178fe40e"}
11:38:53 [11:38:53.156]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHFS9dz/1 with map size 1073741824
11:38:53 [11:38:53.157]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:53 [11:38:53.176]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:080bdc442afc37ed epochNumber=2
{"provingJobId":"1:BASE_PARITY:080bdc442afc37ed"}
11:38:53 [11:38:53.177]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testHFS9dz/2 with map size 1073741824
11:38:53 [11:38:53.179]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:54 [11:38:54.198]
INFO:
prover-client:proving-broker Proving Broker started
11:38:54 [11:38:54.201]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bae1de3d178fe40e settled=false
{"provingJobId":"1:BASE_PARITY:bae1de3d178fe40e","status":"pending"}
11:38:54 [11:38:54.202]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:080bdc442afc37ed settled=false
{"provingJobId":"1:BASE_PARITY:080bdc442afc37ed","status":"pending"}
11:38:54 [11:38:54.203]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bae1de3d178fe40e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bae1de3d178fe40e"}
11:38:54 [11:38:54.203]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bae1de3d178fe40e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bae1de3d178fe40e"}
11:38:54 [11:38:54.225]
INFO:
prover-client:proving-broker Proving Broker started
11:38:54 [11:38:54.226]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:393869469ebdab1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:393869469ebdab1f"}
11:38:54 [11:38:54.227]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testttT35l/1 with map size 1073741824
11:38:54 [11:38:54.229]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:54 [11:38:54.247]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b8f56d36fa72ad2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7b8f56d36fa72ad2"}
11:38:54 [11:38:54.248]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testttT35l/2 with map size 1073741824
11:38:54 [11:38:54.249]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:54 [11:38:54.265]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:393869469ebdab1f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:393869469ebdab1f"}
11:38:54 [11:38:54.272]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7b8f56d36fa72ad2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7b8f56d36fa72ad2"}
11:38:54 [11:38:54.284]
INFO:
prover-client:proving-broker Proving Broker started
11:38:54 [11:38:54.285]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:675104ebfcb03664 epochNumber=1
{"provingJobId":"1:BASE_PARITY:675104ebfcb03664"}
11:38:54 [11:38:54.288]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testj9JNXS/1 with map size 1073741824
11:38:54 [11:38:54.291]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:54 [11:38:54.312]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e37659a8f1e786d0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e37659a8f1e786d0"}
11:38:54 [11:38:54.314]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testj9JNXS/2 with map size 1073741824
11:38:54 [11:38:54.316]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:54 [11:38:54.337]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:675104ebfcb03664 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:675104ebfcb03664"}
11:38:54 [11:38:54.337]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:675104ebfcb03664 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:675104ebfcb03664"}
11:38:54 [11:38:54.345]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e37659a8f1e786d0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e37659a8f1e786d0"}
11:38:54 [11:38:54.356]
INFO:
prover-client:proving-broker Proving Broker started
11:38:54 [11:38:54.357]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:fbce355781135de3 err=test error
{"provingJoId":"1:BASE_PARITY:fbce355781135de3"}
11:38:54 [11:38:54.360]
INFO:
prover-client:proving-broker Proving Broker started
11:38:54 [11:38:54.360]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e7da8fe4f13014d1 not found
{"provingJobId":"1:BASE_PARITY:e7da8fe4f13014d1"}
11:38:54 [11:38:54.364]
INFO:
prover-client:proving-broker Proving Broker started
11:38:54 [11:38:54.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61308887f72c252c epochNumber=1
{"provingJobId":"1:BASE_PARITY:61308887f72c252c"}
11:38:54 [11:38:54.366]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIyuhqC/1 with map size 1073741824
11:38:54 [11:38:54.368]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:54 [11:38:54.392]
INFO:
prover-client:proving-broker Proving Broker started
11:38:54 [11:38:54.393]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02aaed0161b295c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:02aaed0161b295c3"}
11:38:54 [11:38:54.394]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfyrf1n/1 with map size 1073741824
11:38:54 [11:38:54.396]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:54 [11:38:54.544]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:02aaed0161b295c3 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:02aaed0161b295c3"}
11:38:54 [11:38:54.555]
INFO:
prover-client:proving-broker Proving Broker started
11:38:54 [11:38:54.556]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:30db25adc9762532 epochNumber=1
{"provingJobId":"1:BASE_PARITY:30db25adc9762532"}
11:38:54 [11:38:54.557]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxWg2wY/1 with map size 1073741824
11:38:54 [11:38:54.559]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:54 [11:38:54.686]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:30db25adc9762532 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:30db25adc9762532"}
11:38:54 [11:38:54.690]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f07a5d3c76686d16 epochNumber=10
{"provingJobId":"1:BASE_PARITY:f07a5d3c76686d16"}
11:38:54 [11:38:54.692]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testxWg2wY/10 with map size 1073741824
11:38:54 [11:38:54.695]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:54 [11:38:54.739]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:54 [11:38:54.740]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
11:38:54 [11:38:54.854]
INFO:
prover-client:proving-broker Proving Broker started
11:38:54 [11:38:54.855]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40c15bed2357153b epochNumber=1
{"provingJobId":"1:BASE_PARITY:40c15bed2357153b"}
11:38:54 [11:38:54.857]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAy2DtZ/1 with map size 1073741824
11:38:54 [11:38:54.859]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.031]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:40c15bed2357153b timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:40c15bed2357153b"}
11:38:55 [11:38:55.085]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fda38849cbe8a47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fda38849cbe8a47"}
11:38:55 [11:38:55.086]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLKzHSe/1 with map size 1073741824
11:38:55 [11:38:55.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.114]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0fda38849cbe8a47 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0fda38849cbe8a47"}
11:38:55 [11:38:55.118]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.118]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87cf2be9597994a2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87cf2be9597994a2"}
11:38:55 [11:38:55.119]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test73eSXN/1 with map size 1073741824
11:38:55 [11:38:55.120]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.141]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:87cf2be9597994a2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:87cf2be9597994a2"}
11:38:55 [11:38:55.142]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:87cf2be9597994a2 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:87cf2be9597994a2"}
11:38:55 [11:38:55.152]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.153]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e99cdba2c46ce730 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e99cdba2c46ce730"}
11:38:55 [11:38:55.154]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYJ35I6/1 with map size 1073741824
11:38:55 [11:38:55.158]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.175]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e99cdba2c46ce730 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e99cdba2c46ce730"}
11:38:55 [11:38:55.186]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.187]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2fafa8e5c789b04e epochNumber=1
{"provingJobId":"1:BASE_PARITY:2fafa8e5c789b04e"}
11:38:55 [11:38:55.188]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testi5sF4A/1 with map size 1073741824
11:38:55 [11:38:55.191]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.209]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2fafa8e5c789b04e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:2fafa8e5c789b04e"}
11:38:55 [11:38:55.211]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:258651be7a4ad6b0 epochNumber=3
{"provingJobId":"1:BASE_PARITY:258651be7a4ad6b0"}
11:38:55 [11:38:55.212]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testi5sF4A/3 with map size 1073741824
11:38:55 [11:38:55.214]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.237]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:55 [11:38:55.238]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
11:38:55 [11:38:55.257]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2fafa8e5c789b04e err=test error
{"provingJoId":"1:BASE_PARITY:2fafa8e5c789b04e"}
11:38:55 [11:38:55.265]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvgVgn6/1 with map size 1073741824
11:38:55 [11:38:55.269]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.294]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvgVgn6/2 with map size 1073741824
11:38:55 [11:38:55.297]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.322]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.325]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:95f15fbf4a115a38 settled=false
{"provingJobId":"1:BASE_PARITY:95f15fbf4a115a38","status":"pending"}
11:38:55 [11:38:55.327]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b6604dedfce2ec42 settled=false
{"provingJobId":"1:BASE_PARITY:b6604dedfce2ec42","status":"pending"}
11:38:55 [11:38:55.338]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrKlsft/1 with map size 1073741824
11:38:55 [11:38:55.341]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.359]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrKlsft/2 with map size 1073741824
11:38:55 [11:38:55.364]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.397]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.401]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d1e57d2a502caecb settled=true
{"provingJobId":"1:BASE_PARITY:d1e57d2a502caecb","status":"fulfilled"}
11:38:55 [11:38:55.403]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:6073cf66b3a949c3 settled=true
{"provingJobId":"2:BASE_PARITY:6073cf66b3a949c3","status":"fulfilled"}
11:38:55 [11:38:55.412]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNpvw6v/1 with map size 1073741824
11:38:55 [11:38:55.416]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.444]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNpvw6v/2 with map size 1073741824
11:38:55 [11:38:55.446]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.464]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.466]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b6cc5b3b49474bc5 settled=true
{"provingJobId":"1:BASE_PARITY:b6cc5b3b49474bc5","status":"fulfilled"}
11:38:55 [11:38:55.468]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:57f88b33f57248ae settled=false
{"provingJobId":"1:BASE_PARITY:57f88b33f57248ae","status":"pending"}
11:38:55 [11:38:55.476]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.477]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f38d24439180cd32 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f38d24439180cd32"}
11:38:55 [11:38:55.478]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzE9IBi/1 with map size 1073741824
11:38:55 [11:38:55.481]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.503]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.503]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40afc9d977965286 epochNumber=1
{"provingJobId":"1:BASE_PARITY:40afc9d977965286"}
11:38:55 [11:38:55.504]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:40afc9d977965286: Error: db error: Error: db error
11:38:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:40afc9d977965286"}
11:38:55 [11:38:55.512]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6edeb79217deace7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6edeb79217deace7"}
11:38:55 [11:38:55.515]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqi6Oze/1 with map size 1073741824
11:38:55 [11:38:55.520]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.546]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6edeb79217deace7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6edeb79217deace7"}
11:38:55 [11:38:55.547]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6edeb79217deace7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6edeb79217deace7"}
11:38:55 [11:38:55.560]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.560]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a57a67aed17f5178 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a57a67aed17f5178"}
11:38:55 [11:38:55.561]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testm7xtLW/1 with map size 1073741824
11:38:55 [11:38:55.567]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.585]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a57a67aed17f5178 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a57a67aed17f5178"}
11:38:55 [11:38:55.585]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a57a67aed17f5178 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a57a67aed17f5178"}
11:38:55 [11:38:55.586]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:a57a67aed17f5178: Error: db error
11:38:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:a57a67aed17f5178"}
11:38:55 [11:38:55.593]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99c2d3f0b40e055f epochNumber=1
{"provingJobId":"1:BASE_PARITY:99c2d3f0b40e055f"}
11:38:55 [11:38:55.595]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testw3D239/1 with map size 1073741824
11:38:55 [11:38:55.599]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.622]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:99c2d3f0b40e055f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:99c2d3f0b40e055f"}
11:38:55 [11:38:55.622]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:99c2d3f0b40e055f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:99c2d3f0b40e055f"}
11:38:55 [11:38:55.636]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.637]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21afa0e7f6176bb9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:21afa0e7f6176bb9"}
11:38:55 [11:38:55.638]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHjnPeo/1 with map size 1073741824
11:38:55 [11:38:55.645]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.667]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:21afa0e7f6176bb9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:21afa0e7f6176bb9"}
11:38:55 [11:38:55.668]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:21afa0e7f6176bb9 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:21afa0e7f6176bb9"}
11:38:55 [11:38:55.669]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:21afa0e7f6176bb9 jobErr=test error: Error: db error
11:38:55 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:21afa0e7f6176bb9"}
11:38:55 [11:38:55.675]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.676]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d39c66cb49b98d5e not found
{"provingJobId":"1:BASE_PARITY:d39c66cb49b98d5e"}
11:38:55 [11:38:55.680]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.681]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bc1d511c79871180 err=test error
{"provingJoId":"1:BASE_PARITY:bc1d511c79871180"}
11:38:55 [11:38:55.684]
INFO:
prover-client:proving-broker Proving Broker started
11:38:55 [11:38:55.695]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9567e333c0f691d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9567e333c0f691d"}
11:38:55 [11:38:55.697]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvjpdES/1 with map size 1073741824
11:38:55 [11:38:55.701]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.721]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b9567e333c0f691d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b9567e333c0f691d"}
11:38:55 [11:38:55.722]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b9567e333c0f691d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b9567e333c0f691d"}
11:38:55 [11:38:55.739]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:c6e2381ef75571e5 epochNumber=2
{"provingJobId":"2:BASE_PARITY:c6e2381ef75571e5"}
11:38:55 [11:38:55.741]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvjpdES/2 with map size 1073741824
11:38:55 [11:38:55.742]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.766]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:c6e2381ef75571e5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:c6e2381ef75571e5"}
11:38:55 [11:38:55.766]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c6e2381ef75571e5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:c6e2381ef75571e5"}
11:38:55 [11:38:55.835]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:8a098693bb67610e epochNumber=3
{"provingJobId":"3:BASE_PARITY:8a098693bb67610e"}
11:38:55 [11:38:55.836]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testvjpdES/3 with map size 1073741824
11:38:55 [11:38:55.840]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.864]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:55 [11:38:55.865]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
11:38:55 [11:38:55.920]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:80d4678caad380c5 epochNumber=4
{"provingJobId":"4:BASE_PARITY:80d4678caad380c5"}
11:38:55 [11:38:55.921]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testvjpdES/4 with map size 1073741824
11:38:55 [11:38:55.927]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:55 [11:38:55.971]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:55 [11:38:55.972]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
11:38:56 [11:38:56.011]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:1483afc93756d72c epochNumber=5
{"provingJobId":"5:BASE_PARITY:1483afc93756d72c"}
11:38:56 [11:38:56.012]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testvjpdES/5 with map size 1073741824
11:38:56 [11:38:56.014]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:38:56 [11:38:56.052]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:38:56 [11:38:56.053]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
11:38:56
PASS src/proving_broker/proving_broker.test.ts (
13.157 s)
11:38:56 ProvingBroker
11:38:56 Producer API
11:38:56
✓ refuses stale jobs (35 ms)
11:38:56
✓ enqueues jobs (5 ms)
11:38:56
✓ ignores duplicate jobs (5 ms)
11:38:56
✓ reports correct status when enqueuing repeat jobs (5 ms)
11:38:56
✓ throws an error in case of duplicate job IDs (21 ms)
11:38:56
✓ returns not-found status for non-existing jobs (2 ms)
11:38:56
✓ cancels jobs in queue (3 ms)
11:38:56
✓ cancels jobs in-progress (2 ms)
11:38:56
✓ returns job result if successful (3 ms)
11:38:56
✓ returns job error if failed (1 ms)
11:38:56
✓ correctly returns job status for concurrent writes (2 ms)
11:38:56
✓ refuses stale jobs (35 ms)
11:38:56
✓ enqueues jobs (29 ms)
11:38:56
✓ ignores duplicate jobs (32 ms)
11:38:56
✓ reports correct status when enqueuing repeat jobs (33 ms)
11:38:56
✓ throws an error in case of duplicate job IDs (29 ms)
11:38:56
✓ returns not-found status for non-existing jobs (2 ms)
11:38:56
✓ cancels jobs in queue (41 ms)
11:38:56
✓ cancels jobs in-progress (29 ms)
11:38:56
✓ returns job result if successful (42 ms)
11:38:56
✓ returns job error if failed (46 ms)
11:38:56
✓ correctly returns job status for concurrent writes (41 ms)
11:38:56 Consumer API
11:38:56
✓ returns undefined if no jobs are available (8 ms)
11:38:56
✓ returns jobs in priority order (2 ms)
11:38:56
✓ returns undefined if no jobs are available for the given allowList (1 ms)
11:38:56
✓ returns a job if it is in the allowList (1 ms)
11:38:56
✓ returns the most important job if it is in the allowList (2 ms)
11:38:56
✓ returns any job if filter is empty (3 ms)
11:38:56
✓ returns a new job when reporting job success (2 ms)
11:38:56
✓ returns a new job when reporting permanent error (1 ms)
11:38:56
✓ returns a new job when reporting retry-able error (2 ms)
11:38:56
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
11:38:56
✓ returns a new job if job is already in progress elsewhere (1056 ms)
11:38:56
✓ avoids sending the same job to a new agent after a restart (1004 ms)
11:38:56
✓ avoids sending a completed job to a new agent after a restart (1004 ms)
11:38:56
✓ tracks job result if in progress (3 ms)
11:38:56
✓ tracks job result even if job is in queue (2 ms)
11:38:56
✓ ignores reported job error if unknown job (1 ms)
11:38:56
✓ ignores job result if unknown job (1 ms)
11:38:56
✓ returns undefined if no jobs are available (2 ms)
11:38:56
✓ returns jobs in priority order (59 ms)
11:38:56
✓ returns undefined if no jobs are available for the given allowList (20 ms)
11:38:56
✓ returns a job if it is in the allowList (70 ms)
11:38:56
✓ returns the most important job if it is in the allowList (80 ms)
11:38:56
✓ returns any job if filter is empty (72 ms)
11:38:56
✓ returns a new job when reporting job success (47 ms)
11:38:56
✓ returns a new job when reporting permanent error (44 ms)
11:38:56
✓ returns a new job when reporting retry-able error (41 ms)
11:38:56
✓ returns a new job when reporting progress if current one is cancelled (41 ms)
11:38:56
✓ returns a new job if job is already in progress elsewhere (1115 ms)
11:38:56
✓ avoids sending the same job to a new agent after a restart (1060 ms)
11:38:56
✓ avoids sending a completed job to a new agent after a restart (1065 ms)
11:38:56
✓ tracks job result if in progress (61 ms)
11:38:56
✓ tracks job result even if job is in queue (74 ms)
11:38:56
✓ ignores reported job error if unknown job (4 ms)
11:38:56
✓ ignores job result if unknown job (3 ms)
11:38:56 Timeouts
11:38:56
✓ tracks in progress jobs (1 ms)
11:38:56
✓ re-enqueues jobs that time out (108 ms)
11:38:56
✓ cancel stale jobs that time out (231 ms)
11:38:56
✓ keeps the jobs in progress while it is alive (203 ms)
11:38:56
✓ tracks in progress jobs (28 ms)
11:38:56
✓ re-enqueues jobs that time out (161 ms)
11:38:56
✓ cancel stale jobs that time out (300 ms)
11:38:56
✓ keeps the jobs in progress while it is alive (229 ms)
11:38:56 Retries
11:38:56
✓ retries jobs (4 ms)
11:38:56
✓ retries up to a maximum number of times (3 ms)
11:38:56
✓ passing retry=false does not retry (2 ms)
11:38:56
✓ does not retry if job is stale (30 ms)
11:38:56
✓ retries jobs (34 ms)
11:38:56
✓ retries up to a maximum number of times (34 ms)
11:38:56
✓ passing retry=false does not retry (32 ms)
11:38:56
✓ does not retry if job is stale (77 ms)
11:38:56 Database management
11:38:56
✓ re-enqueues proof requests on start (4 ms)
11:38:56
✓ restores proof results on start (3 ms)
11:38:56
✓ only re-enqueues unfinished jobs (2 ms)
11:38:56
✓ saves job when enqueued (3 ms)
11:38:56
✓ does not retain job if database fails to save (2 ms)
11:38:56
✓ saves job result (1 ms)
11:38:56
✓ saves result even if database fails to save (2 ms)
11:38:56
✓ saves job error (2 ms)
11:38:56
✓ saves job error even if database fails to save (2 ms)
11:38:56
✓ does not save job result if job is unknown (2 ms)
11:38:56
✓ does not save job error if job is unknown (1 ms)
11:38:56
✓ cleans up old jobs periodically (265 ms)
11:38:56
✓ re-enqueues proof requests on start (71 ms)
11:38:56
✓ restores proof results on start (74 ms)
11:38:56
✓ only re-enqueues unfinished jobs (65 ms)
11:38:56
✓ saves job when enqueued (29 ms)
11:38:56
✓ does not retain job if database fails to save (4 ms)
11:38:56
✓ saves job result (50 ms)
11:38:56
✓ saves result even if database fails to save (33 ms)
11:38:56
✓ saves job error (43 ms)
11:38:56
✓ saves job error even if database fails to save (38 ms)
11:38:56
✓ does not save job result if job is unknown (5 ms)
11:38:56
✓ does not save job error if job is unknown (4 ms)
11:38:56
✓ cleans up old jobs periodically (405 ms)
11:38:56
11:38:56
Test Suites: 1 passed, 1 total
11:38:56
Tests: 96 passed, 96 total
11:38:56
Snapshots: 0 total
11:38:56
Time: 13.28 s
11:38:56
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
11:38:56
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?