Parent Log:
http://ci.aztec-labs.com/03b9ba3e2cc99f2a
Command: 1fec7f5307cfdb51 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/7e505bcd7fcd90a7d5fe893194272157cc9ec848
Env: REF_NAME=gh-readonly-queue/next/pr-15159-88a99fb5fb938035dfeff0d9ed0cfd81b9eb2c5a CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Thu Jun 19 16:35:00 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15159_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
16:35:05 [16:35:05.024]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.032]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2efc619d0ce892f epochNumber=42
{"provingJobId":"1:BASE_PARITY:d2efc619d0ce892f"}
16:35:05 [16:35:05.036]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:f370e8201f5d9b7b epochNumber=1
{"provingJobId":"1:BASE_PARITY:f370e8201f5d9b7b"}
16:35:05 [16:35:05.076]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc7b30e1bec7afec epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc7b30e1bec7afec"}
16:35:05 [16:35:05.078]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d02c138abd39e05a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d02c138abd39e05a"}
16:35:05 [16:35:05.081]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb91ea05d7bec62a epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb91ea05d7bec62a"}
16:35:05 [16:35:05.083]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:bb91ea05d7bec62a epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:bb91ea05d7bec62a"}
16:35:05 [16:35:05.086]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe6d49d89c69949f epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe6d49d89c69949f"}
16:35:05 [16:35:05.088]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fe6d49d89c69949f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fe6d49d89c69949f"}
16:35:05 [16:35:05.089]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fe6d49d89c69949f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fe6d49d89c69949f"}
16:35:05 [16:35:05.089]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fe6d49d89c69949f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fe6d49d89c69949f"}
16:35:05 [16:35:05.091]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.091]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d6f585ead7fa1c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d6f585ead7fa1c4"}
16:35:05 [16:35:05.121]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.123]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0cbe41878947cd1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0cbe41878947cd1"}
16:35:05 [16:35:05.125]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b0cbe41878947cd1
{"provingJobId":"1:BASE_PARITY:b0cbe41878947cd1"}
16:35:05 [16:35:05.126]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b0cbe41878947cd1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b0cbe41878947cd1"}
16:35:05 [16:35:05.127]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b0cbe41878947cd1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b0cbe41878947cd1"}
16:35:05 [16:35:05.129]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.129]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ba07b985ba7c2e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ba07b985ba7c2e0"}
16:35:05 [16:35:05.130]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2ba07b985ba7c2e0
{"provingJobId":"1:BASE_PARITY:2ba07b985ba7c2e0"}
16:35:05 [16:35:05.131]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2ba07b985ba7c2e0 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2ba07b985ba7c2e0"}
16:35:05 [16:35:05.132]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd96115f93743124 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd96115f93743124"}
16:35:05 [16:35:05.134]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dd96115f93743124 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dd96115f93743124"}
16:35:05 [16:35:05.134]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dd96115f93743124 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dd96115f93743124"}
16:35:05 [16:35:05.136]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.136]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b09fcce4d6fb78ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:b09fcce4d6fb78ea"}
16:35:05 [16:35:05.137]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b09fcce4d6fb78ea type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b09fcce4d6fb78ea"}
16:35:05 [16:35:05.137]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b09fcce4d6fb78ea type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b09fcce4d6fb78ea"}
16:35:05 [16:35:05.138]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.139]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed8f6c38cd2858d7 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ed8f6c38cd2858d7"}
16:35:05 [16:35:05.139]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ed8f6c38cd2858d7 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ed8f6c38cd2858d7"}
16:35:05 [16:35:05.140]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab9e01f9803bc004 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ab9e01f9803bc004"}
16:35:05 [16:35:05.140]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ed8f6c38cd2858d7 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ed8f6c38cd2858d7"}
16:35:05 [16:35:05.140]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a577cdc63bdbb21 epochNumber=0
{"provingJobId":"1:BASE_PARITY:9a577cdc63bdbb21"}
16:35:05 [16:35:05.142]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.152]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.153]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7860adc02d6c7b9c epochNumber=2
{"provingJobId":"1:BASE_PARITY:7860adc02d6c7b9c"}
16:35:05 [16:35:05.153]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec9f1a74efd012ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec9f1a74efd012ca"}
16:35:05 [16:35:05.154]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.155]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:816f9a5c80df2616 epochNumber=1
{"provingJobId":"1:BASE_PARITY:816f9a5c80df2616"}
16:35:05 [16:35:05.156]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.156]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16262b52a09d7387 epochNumber=1
{"provingJobId":"1:BASE_PARITY:16262b52a09d7387"}
16:35:05 [16:35:05.157]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61bcd831876f83cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:61bcd831876f83cd"}
16:35:05 [16:35:05.157]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:efb0ec771f6cdaa7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:efb0ec771f6cdaa7"}
16:35:05 [16:35:05.157]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1638c3305828827c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1638c3305828827c"}
16:35:05 [16:35:05.158]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d5508baa949bdec7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d5508baa949bdec7"}
16:35:05 [16:35:05.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a6eae7dbf74e9d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a6eae7dbf74e9d7"}
16:35:05 [16:35:05.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb034dd221c80bf3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fb034dd221c80bf3"}
16:35:05 [16:35:05.160]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2b2e59a4b973bf1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c2b2e59a4b973bf1"}
16:35:05 [16:35:05.161]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.161]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13af3c6d82c9e923 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13af3c6d82c9e923"}
16:35:05 [16:35:05.161]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7dfff8b3e583a30d epochNumber=1
{"provingJobId":"1:BASE_PARITY:7dfff8b3e583a30d"}
16:35:05 [16:35:05.162]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2790394ec8a9e72f epochNumber=2
{"provingJobId":"1:BASE_PARITY:2790394ec8a9e72f"}
16:35:05 [16:35:05.162]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:becd462232b366e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:becd462232b366e9"}
16:35:05 [16:35:05.163]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.164]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97bd41774ef2ae81 epochNumber=1
{"provingJobId":"1:BASE_PARITY:97bd41774ef2ae81"}
16:35:05 [16:35:05.164]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86a4fcd189d8263e epochNumber=1
{"provingJobId":"1:BASE_PARITY:86a4fcd189d8263e"}
16:35:05 [16:35:05.164]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:97bd41774ef2ae81 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:97bd41774ef2ae81"}
16:35:05 [16:35:05.166]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.166]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c54962f009ac03b epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c54962f009ac03b"}
16:35:05 [16:35:05.167]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f7fb1630ba092ef epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f7fb1630ba092ef"}
16:35:05 [16:35:05.167]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5c54962f009ac03b type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:5c54962f009ac03b"}
16:35:05 [16:35:05.168]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b4c7b1065417d4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b4c7b1065417d4b"}
16:35:05 [16:35:05.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26af70f38655cb91 epochNumber=1
{"provingJobId":"1:BASE_PARITY:26af70f38655cb91"}
16:35:05 [16:35:05.169]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:7b4c7b1065417d4b type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:7b4c7b1065417d4b"}
16:35:05 [16:35:05.170]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.170]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a28c3e315641829f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a28c3e315641829f"}
16:35:05 [16:35:05.171]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a28c3e315641829f
{"provingJobId":"1:BASE_PARITY:a28c3e315641829f"}
16:35:05 [16:35:05.171]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a28c3e315641829f type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a28c3e315641829f"}
16:35:05 [16:35:05.171]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd4f6efaafe7b844 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd4f6efaafe7b844"}
16:35:05 [16:35:05.172]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a28c3e315641829f has already been completed
{"provingJobId":"1:BASE_PARITY:a28c3e315641829f"}
16:35:05 [16:35:05.173]
INFO:
prover-client:proving-broker Proving Broker started
16:35:05 [16:35:05.173]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba59cd3ac5c018c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba59cd3ac5c018c3"}
16:35:05 [16:35:05.173]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e6f60be1cb0cc15 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6e6f60be1cb0cc15"}
16:35:06 [16:35:06.226]
INFO:
prover-client:proving-broker Proving Broker started
16:35:06 [16:35:06.227]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ba59cd3ac5c018c3 settled=false
{"provingJobId":"1:BASE_PARITY:ba59cd3ac5c018c3","status":"pending"}
16:35:06 [16:35:06.228]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6e6f60be1cb0cc15 settled=false
{"provingJobId":"1:BASE_PARITY:6e6f60be1cb0cc15","status":"pending"}
16:35:06 [16:35:06.228]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ba59cd3ac5c018c3 type=BASE_PARITY startedAt=1750350905174 older agent has taken job
{"provingJobId":"1:BASE_PARITY:ba59cd3ac5c018c3"}
16:35:06 [16:35:06.229]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ba59cd3ac5c018c3 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:ba59cd3ac5c018c3"}
16:35:06 [16:35:06.231]
INFO:
prover-client:proving-broker Proving Broker started
16:35:06 [16:35:06.231]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b7ab584639d7826 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b7ab584639d7826"}
16:35:06 [16:35:06.231]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84618ea4f522086a epochNumber=2
{"provingJobId":"1:BASE_PARITY:84618ea4f522086a"}
16:35:07 [16:35:07.232]
INFO:
prover-client:proving-broker Proving Broker started
16:35:07 [16:35:07.234]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3b7ab584639d7826 settled=false
{"provingJobId":"1:BASE_PARITY:3b7ab584639d7826","status":"pending"}
16:35:07 [16:35:07.234]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:84618ea4f522086a settled=false
{"provingJobId":"1:BASE_PARITY:84618ea4f522086a","status":"pending"}
16:35:07 [16:35:07.235]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3b7ab584639d7826 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:3b7ab584639d7826"}
16:35:07 [16:35:07.236]
INFO:
prover-client:proving-broker Proving Broker started
16:35:07 [16:35:07.237]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8100d4aaa58f874a epochNumber=1
{"provingJobId":"1:BASE_PARITY:8100d4aaa58f874a"}
16:35:07 [16:35:07.237]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:feeaef920c77435a epochNumber=2
{"provingJobId":"1:BASE_PARITY:feeaef920c77435a"}
16:35:08 [16:35:08.238]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.238]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8100d4aaa58f874a settled=false
{"provingJobId":"1:BASE_PARITY:8100d4aaa58f874a","status":"pending"}
16:35:08 [16:35:08.239]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:feeaef920c77435a settled=false
{"provingJobId":"1:BASE_PARITY:feeaef920c77435a","status":"pending"}
16:35:08 [16:35:08.239]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8100d4aaa58f874a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8100d4aaa58f874a"}
16:35:08 [16:35:08.239]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8100d4aaa58f874a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8100d4aaa58f874a"}
16:35:08 [16:35:08.241]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.241]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de2fcc20d79e8f7c epochNumber=1
{"provingJobId":"1:BASE_PARITY:de2fcc20d79e8f7c"}
16:35:08 [16:35:08.241]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f69918c6048522ca epochNumber=2
{"provingJobId":"1:BASE_PARITY:f69918c6048522ca"}
16:35:08 [16:35:08.242]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:de2fcc20d79e8f7c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:de2fcc20d79e8f7c"}
16:35:08 [16:35:08.242]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f69918c6048522ca type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f69918c6048522ca"}
16:35:08 [16:35:08.243]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.244]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55a7dada1dcbf7da epochNumber=1
{"provingJobId":"1:BASE_PARITY:55a7dada1dcbf7da"}
16:35:08 [16:35:08.244]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3b1fef1bbfb5a69 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f3b1fef1bbfb5a69"}
16:35:08 [16:35:08.244]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:55a7dada1dcbf7da type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:55a7dada1dcbf7da"}
16:35:08 [16:35:08.244]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:55a7dada1dcbf7da type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:55a7dada1dcbf7da"}
16:35:08 [16:35:08.245]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f3b1fef1bbfb5a69 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f3b1fef1bbfb5a69"}
16:35:08 [16:35:08.245]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.246]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7b9b60f9d43a816a err=test error
{"provingJoId":"1:BASE_PARITY:7b9b60f9d43a816a"}
16:35:08 [16:35:08.247]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.248]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:100a267fab800e0a not found
{"provingJobId":"1:BASE_PARITY:100a267fab800e0a"}
16:35:08 [16:35:08.249]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c42f2fafcf0dc360 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c42f2fafcf0dc360"}
16:35:08 [16:35:08.251]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.251]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1052b58b85b5ef89 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1052b58b85b5ef89"}
16:35:08 [16:35:08.354]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1052b58b85b5ef89 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1052b58b85b5ef89"}
16:35:08 [16:35:08.360]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.361]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:452c2dd11e6687df epochNumber=1
{"provingJobId":"1:BASE_PARITY:452c2dd11e6687df"}
16:35:08 [16:35:08.461]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:452c2dd11e6687df timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:452c2dd11e6687df"}
16:35:08 [16:35:08.463]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1693080e181cd86c epochNumber=10
{"provingJobId":"1:BASE_PARITY:1693080e181cd86c"}
16:35:08 [16:35:08.487]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:08 [16:35:08.590]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.591]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75c4230bf41f35b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:75c4230bf41f35b4"}
16:35:08 [16:35:08.743]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:75c4230bf41f35b4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:75c4230bf41f35b4"}
16:35:08 [16:35:08.795]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.796]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d967628d09c608cc epochNumber=1
{"provingJobId":"1:BASE_PARITY:d967628d09c608cc"}
16:35:08 [16:35:08.797]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d967628d09c608cc type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d967628d09c608cc"}
16:35:08 [16:35:08.798]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.798]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a445b876402e9c8e epochNumber=1
{"provingJobId":"1:BASE_PARITY:a445b876402e9c8e"}
16:35:08 [16:35:08.799]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a445b876402e9c8e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a445b876402e9c8e"}
16:35:08 [16:35:08.799]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a445b876402e9c8e type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:a445b876402e9c8e"}
16:35:08 [16:35:08.800]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.800]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66cc7bc1aafa339a epochNumber=1
{"provingJobId":"1:BASE_PARITY:66cc7bc1aafa339a"}
16:35:08 [16:35:08.801]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:66cc7bc1aafa339a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:66cc7bc1aafa339a"}
16:35:08 [16:35:08.801]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb22e90d6a8373cc epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb22e90d6a8373cc"}
16:35:08 [16:35:08.802]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:eb22e90d6a8373cc type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:eb22e90d6a8373cc"}
16:35:08 [16:35:08.803]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12a8292e2fef971f epochNumber=3
{"provingJobId":"1:BASE_PARITY:12a8292e2fef971f"}
16:35:08 [16:35:08.827]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:08 [16:35:08.828]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:eb22e90d6a8373cc err=test error
{"provingJoId":"1:BASE_PARITY:eb22e90d6a8373cc"}
16:35:08 [16:35:08.830]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.830]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7fcfda0bb6824da8 settled=false
{"provingJobId":"1:BASE_PARITY:7fcfda0bb6824da8","status":"pending"}
16:35:08 [16:35:08.830]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d9c90c71eaa8b57b settled=false
{"provingJobId":"1:BASE_PARITY:d9c90c71eaa8b57b","status":"pending"}
16:35:08 [16:35:08.833]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.833]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6058933bba1b53aa settled=true
{"provingJobId":"1:BASE_PARITY:6058933bba1b53aa","status":"fulfilled"}
16:35:08 [16:35:08.834]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:a079a7d3105200c2 settled=true
{"provingJobId":"2:BASE_PARITY:a079a7d3105200c2","status":"fulfilled"}
16:35:08 [16:35:08.835]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.835]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9e000fc803bf0cae settled=true
{"provingJobId":"1:BASE_PARITY:9e000fc803bf0cae","status":"fulfilled"}
16:35:08 [16:35:08.836]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9f46acafdc08e059 settled=false
{"provingJobId":"1:BASE_PARITY:9f46acafdc08e059","status":"pending"}
16:35:08 [16:35:08.837]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.838]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6cc95d9364fffb22 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6cc95d9364fffb22"}
16:35:08 [16:35:08.840]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.840]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa7abdde9b395176 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa7abdde9b395176"}
16:35:08 [16:35:08.841]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:aa7abdde9b395176: Error: db error: Error: db error
16:35:08 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:aa7abdde9b395176"}
16:35:08 [16:35:08.842]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.842]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cdd092dfef8d65aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:cdd092dfef8d65aa"}
16:35:08 [16:35:08.843]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cdd092dfef8d65aa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:cdd092dfef8d65aa"}
16:35:08 [16:35:08.843]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cdd092dfef8d65aa type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cdd092dfef8d65aa"}
16:35:08 [16:35:08.844]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.844]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc96079a67b30d20 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc96079a67b30d20"}
16:35:08 [16:35:08.844]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fc96079a67b30d20 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fc96079a67b30d20"}
16:35:08 [16:35:08.844]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fc96079a67b30d20 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fc96079a67b30d20"}
16:35:08 [16:35:08.845]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:fc96079a67b30d20: Error: db error
16:35:08 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:fc96079a67b30d20"}
16:35:08 [16:35:08.846]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f38ca78ab82ce9e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f38ca78ab82ce9e7"}
16:35:08 [16:35:08.846]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f38ca78ab82ce9e7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f38ca78ab82ce9e7"}
16:35:08 [16:35:08.847]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f38ca78ab82ce9e7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f38ca78ab82ce9e7"}
16:35:08 [16:35:08.848]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.848]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb121a93cd3735cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb121a93cd3735cd"}
16:35:08 [16:35:08.849]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fb121a93cd3735cd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fb121a93cd3735cd"}
16:35:08 [16:35:08.849]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fb121a93cd3735cd type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fb121a93cd3735cd"}
16:35:08 [16:35:08.850]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:fb121a93cd3735cd jobErr=test error: Error: db error
16:35:08 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:fb121a93cd3735cd"}
16:35:08 [16:35:08.851]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.852]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8202a2d80c52c5d0 not found
{"provingJobId":"1:BASE_PARITY:8202a2d80c52c5d0"}
16:35:08 [16:35:08.853]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.854]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:eb8765bd060df0fa err=test error
{"provingJoId":"1:BASE_PARITY:eb8765bd060df0fa"}
16:35:08 [16:35:08.855]
INFO:
prover-client:proving-broker Proving Broker started
16:35:08 [16:35:08.867]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d87b21f0657f535f epochNumber=1
{"provingJobId":"1:BASE_PARITY:d87b21f0657f535f"}
16:35:08 [16:35:08.867]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d87b21f0657f535f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d87b21f0657f535f"}
16:35:08 [16:35:08.868]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d87b21f0657f535f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d87b21f0657f535f"}
16:35:08 [16:35:08.878]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:7a89660e66413e65 epochNumber=2
{"provingJobId":"2:BASE_PARITY:7a89660e66413e65"}
16:35:08 [16:35:08.878]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:7a89660e66413e65 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:7a89660e66413e65"}
16:35:08 [16:35:08.879]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:7a89660e66413e65 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:7a89660e66413e65"}
16:35:08 [16:35:08.940]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:f8a73e9cbb51d3b0 epochNumber=3
{"provingJobId":"3:BASE_PARITY:f8a73e9cbb51d3b0"}
16:35:08 [16:35:08.957]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:09 [16:35:09.004]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:58fd04caeba9d718 epochNumber=4
{"provingJobId":"4:BASE_PARITY:58fd04caeba9d718"}
16:35:09 [16:35:09.009]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:09 [16:35:09.066]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:5f92ac729dc9d200 epochNumber=5
{"provingJobId":"5:BASE_PARITY:5f92ac729dc9d200"}
16:35:09 [16:35:09.084]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:09 [16:35:09.123]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.125]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8bb744dbabc0d434 epochNumber=42
{"provingJobId":"1:BASE_PARITY:8bb744dbabc0d434"}
16:35:09 [16:35:09.126]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testkwhCql/42 with map size 1073741824
16:35:09 [16:35:09.130]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.160]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:a6f0522f46f4d3f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6f0522f46f4d3f8"}
16:35:09 [16:35:09.167]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea62c03597cfceda epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea62c03597cfceda"}
16:35:09 [16:35:09.169]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYZ5ROA/1 with map size 1073741824
16:35:09 [16:35:09.171]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.190]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86ba953795fcedd6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:86ba953795fcedd6"}
16:35:09 [16:35:09.204]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.205]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:658e0c8657cd30ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:658e0c8657cd30ec"}
16:35:09 [16:35:09.205]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCEjsdu/1 with map size 1073741824
16:35:09 [16:35:09.207]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.232]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:658e0c8657cd30ec epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:658e0c8657cd30ec"}
16:35:09 [16:35:09.237]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.237]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2eaf383d9d8014ba epochNumber=1
{"provingJobId":"1:BASE_PARITY:2eaf383d9d8014ba"}
16:35:09 [16:35:09.238]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOIvLSV/1 with map size 1073741824
16:35:09 [16:35:09.241]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.262]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2eaf383d9d8014ba epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2eaf383d9d8014ba"}
16:35:09 [16:35:09.263]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2eaf383d9d8014ba type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2eaf383d9d8014ba"}
16:35:09 [16:35:09.269]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2eaf383d9d8014ba epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2eaf383d9d8014ba"}
16:35:09 [16:35:09.273]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83c6773a74d6097c epochNumber=1
{"provingJobId":"1:BASE_PARITY:83c6773a74d6097c"}
16:35:09 [16:35:09.274]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPVIUeg/1 with map size 1073741824
16:35:09 [16:35:09.276]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.303]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.308]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.308]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e25a357ab0393ae epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e25a357ab0393ae"}
16:35:09 [16:35:09.309]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1voV7L/1 with map size 1073741824
16:35:09 [16:35:09.311]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.329]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5e25a357ab0393ae
{"provingJobId":"1:BASE_PARITY:5e25a357ab0393ae"}
16:35:09 [16:35:09.329]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5e25a357ab0393ae type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5e25a357ab0393ae"}
16:35:09 [16:35:09.330]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5e25a357ab0393ae type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5e25a357ab0393ae"}
16:35:09 [16:35:09.338]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.339]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe8889e406c46be5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe8889e406c46be5"}
16:35:09 [16:35:09.340]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAqv2So/1 with map size 1073741824
16:35:09 [16:35:09.342]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.359]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:fe8889e406c46be5
{"provingJobId":"1:BASE_PARITY:fe8889e406c46be5"}
16:35:09 [16:35:09.359]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fe8889e406c46be5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:fe8889e406c46be5"}
16:35:09 [16:35:09.368]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.369]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8c25502ed9b7f23 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8c25502ed9b7f23"}
16:35:09 [16:35:09.373]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKha9ok/1 with map size 1073741824
16:35:09 [16:35:09.375]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.393]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b8c25502ed9b7f23 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b8c25502ed9b7f23"}
16:35:09 [16:35:09.394]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b8c25502ed9b7f23 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b8c25502ed9b7f23"}
16:35:09 [16:35:09.403]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.404]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:020e96937d482b5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:020e96937d482b5d"}
16:35:09 [16:35:09.406]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRgeyUc/1 with map size 1073741824
16:35:09 [16:35:09.413]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.435]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:020e96937d482b5d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:020e96937d482b5d"}
16:35:09 [16:35:09.436]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:020e96937d482b5d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:020e96937d482b5d"}
16:35:09 [16:35:09.445]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.446]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af150bfbe4b9be58 epochNumber=0
{"provingJobId":"1:BASE_PARITY:af150bfbe4b9be58"}
16:35:09 [16:35:09.446]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testhRZN8n/0 with map size 1073741824
16:35:09 [16:35:09.448]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.466]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:af150bfbe4b9be58 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:af150bfbe4b9be58"}
16:35:09 [16:35:09.467]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:134bd25b66753756 epochNumber=0
{"provingJobId":"1:BASE_PARITY:134bd25b66753756"}
16:35:09 [16:35:09.467]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:af150bfbe4b9be58 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:af150bfbe4b9be58"}
16:35:09 [16:35:09.468]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e20c55d321839314 epochNumber=0
{"provingJobId":"1:BASE_PARITY:e20c55d321839314"}
16:35:09 [16:35:09.485]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.488]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.488]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf8f59ce99f4367b epochNumber=2
{"provingJobId":"1:BASE_PARITY:cf8f59ce99f4367b"}
16:35:09 [16:35:09.489]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testEhcjAf/2 with map size 1073741824
16:35:09 [16:35:09.491]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf88072024e5a045 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf88072024e5a045"}
16:35:09 [16:35:09.514]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEhcjAf/1 with map size 1073741824
16:35:09 [16:35:09.517]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.540]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40c4f4b395c646b2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:40c4f4b395c646b2"}
16:35:09 [16:35:09.541]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGIDAm4/1 with map size 1073741824
16:35:09 [16:35:09.543]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.564]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.565]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:edb88a2f491be031 epochNumber=1
{"provingJobId":"1:BASE_PARITY:edb88a2f491be031"}
16:35:09 [16:35:09.566]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAwZcn5/1 with map size 1073741824
16:35:09 [16:35:09.568]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.586]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:572be4bcb79eda3f epochNumber=1
{"provingJobId":"1:BASE_PARITY:572be4bcb79eda3f"}
16:35:09 [16:35:09.592]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98f2991b681be509 epochNumber=2
{"provingJobId":"1:BASE_PARITY:98f2991b681be509"}
16:35:09 [16:35:09.592]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAwZcn5/2 with map size 1073741824
16:35:09 [16:35:09.595]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.613]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f75a31c5709ec7a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f75a31c5709ec7a7"}
16:35:09 [16:35:09.622]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.623]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd1084aefecc12ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd1084aefecc12ed"}
16:35:09 [16:35:09.623]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8hZW0a/1 with map size 1073741824
16:35:09 [16:35:09.625]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.643]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea6adf047e40d902 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea6adf047e40d902"}
16:35:09 [16:35:09.649]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5997acb6c5d02612 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5997acb6c5d02612"}
16:35:09 [16:35:09.650]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8hZW0a/2 with map size 1073741824
16:35:09 [16:35:09.652]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.668]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0bbf6e5c4a6a860d epochNumber=1
{"provingJobId":"1:BASE_PARITY:0bbf6e5c4a6a860d"}
16:35:09 [16:35:09.677]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.677]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea40bd606319f62b epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea40bd606319f62b"}
16:35:09 [16:35:09.678]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvycR5C/1 with map size 1073741824
16:35:09 [16:35:09.679]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.696]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a545c4bd918f940 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a545c4bd918f940"}
16:35:09 [16:35:09.702]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:646b28551abb756d epochNumber=2
{"provingJobId":"1:BASE_PARITY:646b28551abb756d"}
16:35:09 [16:35:09.703]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvycR5C/2 with map size 1073741824
16:35:09 [16:35:09.704]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.721]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:474ed7e25b8ea2df epochNumber=1
{"provingJobId":"1:BASE_PARITY:474ed7e25b8ea2df"}
16:35:09 [16:35:09.732]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.733]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12a8c8213feb1537 epochNumber=1
{"provingJobId":"1:BASE_PARITY:12a8c8213feb1537"}
16:35:09 [16:35:09.734]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwV4tlT/1 with map size 1073741824
16:35:09 [16:35:09.736]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.756]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d9c8584e23921e3e epochNumber=1
{"provingJobId":"1:BASE_PARITY:d9c8584e23921e3e"}
16:35:09 [16:35:09.765]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:12a8c8213feb1537 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:12a8c8213feb1537"}
16:35:09 [16:35:09.776]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9df6d44b3423d6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9df6d44b3423d6c"}
16:35:09 [16:35:09.779]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkwxXwh/1 with map size 1073741824
16:35:09 [16:35:09.780]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.797]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:07b29f6eef67bb93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:07b29f6eef67bb93"}
16:35:09 [16:35:09.804]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b9df6d44b3423d6c type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:b9df6d44b3423d6c"}
16:35:09 [16:35:09.813]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:875934ca90ac21eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:875934ca90ac21eb"}
16:35:09 [16:35:09.814]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOHRYZp/1 with map size 1073741824
16:35:09 [16:35:09.816]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.835]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06153a993f27f7a9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:06153a993f27f7a9"}
16:35:09 [16:35:09.841]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:875934ca90ac21eb type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:875934ca90ac21eb"}
16:35:09 [16:35:09.845]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36abc82537309367 epochNumber=1
{"provingJobId":"1:BASE_PARITY:36abc82537309367"}
16:35:09 [16:35:09.846]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5uu6zC/1 with map size 1073741824
16:35:09 [16:35:09.848]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.862]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:36abc82537309367
{"provingJobId":"1:BASE_PARITY:36abc82537309367"}
16:35:09 [16:35:09.863]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:36abc82537309367 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:36abc82537309367"}
16:35:09 [16:35:09.869]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:808791f9948503c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:808791f9948503c3"}
16:35:09 [16:35:09.874]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:36abc82537309367 has already been completed
{"provingJobId":"1:BASE_PARITY:36abc82537309367"}
16:35:09 [16:35:09.878]
INFO:
prover-client:proving-broker Proving Broker started
16:35:09 [16:35:09.878]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ccdc453f3bb1a12 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ccdc453f3bb1a12"}
16:35:09 [16:35:09.879]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGblcDk/1 with map size 1073741824
16:35:09 [16:35:09.880]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:09 [16:35:09.901]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ba36551fece80e1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7ba36551fece80e1"}
16:35:09 [16:35:09.905]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGblcDk/2 with map size 1073741824
16:35:09 [16:35:09.907]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:10 [16:35:10.982]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.989]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5ccdc453f3bb1a12 settled=false
{"provingJobId":"1:BASE_PARITY:5ccdc453f3bb1a12","status":"pending"}
16:35:10 [16:35:10.991]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7ba36551fece80e1 settled=false
{"provingJobId":"1:BASE_PARITY:7ba36551fece80e1","status":"pending"}
16:35:10 [16:35:10.992]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5ccdc453f3bb1a12 type=BASE_PARITY startedAt=1750350909929 older agent has taken job
{"provingJobId":"1:BASE_PARITY:5ccdc453f3bb1a12"}
16:35:10 [16:35:10.993]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5ccdc453f3bb1a12 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:5ccdc453f3bb1a12"}
16:35:10 [16:35:10.997]
INFO:
prover-client:proving-broker Proving Broker started
16:35:10 [16:35:10.998]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93574caad1b17942 epochNumber=1
{"provingJobId":"1:BASE_PARITY:93574caad1b17942"}
16:35:10 [16:35:10.998]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbnhm7f/1 with map size 1073741824
16:35:11 [16:35:11.000]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:11 [16:35:11.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba84b6ffbb014446 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ba84b6ffbb014446"}
16:35:11 [16:35:11.021]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testbnhm7f/2 with map size 1073741824
16:35:11 [16:35:11.024]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:12 [16:35:12.045]
INFO:
prover-client:proving-broker Proving Broker started
16:35:12 [16:35:12.048]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:93574caad1b17942 settled=false
{"provingJobId":"1:BASE_PARITY:93574caad1b17942","status":"pending"}
16:35:12 [16:35:12.050]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ba84b6ffbb014446 settled=false
{"provingJobId":"1:BASE_PARITY:ba84b6ffbb014446","status":"pending"}
16:35:12 [16:35:12.051]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:93574caad1b17942 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:93574caad1b17942"}
16:35:12 [16:35:12.055]
INFO:
prover-client:proving-broker Proving Broker started
16:35:12 [16:35:12.056]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:254c36e7fb9d328e epochNumber=1
{"provingJobId":"1:BASE_PARITY:254c36e7fb9d328e"}
16:35:12 [16:35:12.057]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAf6RN8/1 with map size 1073741824
16:35:12 [16:35:12.059]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:12 [16:35:12.076]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87674dd9a0b6a4db epochNumber=2
{"provingJobId":"1:BASE_PARITY:87674dd9a0b6a4db"}
16:35:12 [16:35:12.077]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAf6RN8/2 with map size 1073741824
16:35:12 [16:35:12.079]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.096]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.098]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:254c36e7fb9d328e settled=false
{"provingJobId":"1:BASE_PARITY:254c36e7fb9d328e","status":"pending"}
16:35:13 [16:35:13.100]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:87674dd9a0b6a4db settled=false
{"provingJobId":"1:BASE_PARITY:87674dd9a0b6a4db","status":"pending"}
16:35:13 [16:35:13.101]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:254c36e7fb9d328e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:254c36e7fb9d328e"}
16:35:13 [16:35:13.101]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:254c36e7fb9d328e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:254c36e7fb9d328e"}
16:35:13 [16:35:13.121]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b9a053d162ea7cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b9a053d162ea7cd"}
16:35:13 [16:35:13.123]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUaR5rv/1 with map size 1073741824
16:35:13 [16:35:13.125]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.142]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47e6af0a8689a2ac epochNumber=2
{"provingJobId":"1:BASE_PARITY:47e6af0a8689a2ac"}
16:35:13 [16:35:13.143]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUaR5rv/2 with map size 1073741824
16:35:13 [16:35:13.145]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.162]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6b9a053d162ea7cd type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6b9a053d162ea7cd"}
16:35:13 [16:35:13.169]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:47e6af0a8689a2ac type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:47e6af0a8689a2ac"}
16:35:13 [16:35:13.180]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.182]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b2101d5add8cbc6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b2101d5add8cbc6"}
16:35:13 [16:35:13.183]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUXkAYA/1 with map size 1073741824
16:35:13 [16:35:13.185]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.201]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:666bf4f65f8fd3b3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:666bf4f65f8fd3b3"}
16:35:13 [16:35:13.202]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUXkAYA/2 with map size 1073741824
16:35:13 [16:35:13.209]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.227]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b2101d5add8cbc6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7b2101d5add8cbc6"}
16:35:13 [16:35:13.228]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7b2101d5add8cbc6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7b2101d5add8cbc6"}
16:35:13 [16:35:13.233]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:666bf4f65f8fd3b3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:666bf4f65f8fd3b3"}
16:35:13 [16:35:13.244]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.245]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f5a49f04c56925ed err=test error
{"provingJoId":"1:BASE_PARITY:f5a49f04c56925ed"}
16:35:13 [16:35:13.249]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.250]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6cbdc8d542adf637 not found
{"provingJobId":"1:BASE_PARITY:6cbdc8d542adf637"}
16:35:13 [16:35:13.254]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9d279e3ab9f7143 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9d279e3ab9f7143"}
16:35:13 [16:35:13.255]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZSOMYd/1 with map size 1073741824
16:35:13 [16:35:13.259]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.285]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d093df03803f443 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2d093df03803f443"}
16:35:13 [16:35:13.287]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1wtO3T/1 with map size 1073741824
16:35:13 [16:35:13.291]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.417]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2d093df03803f443 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2d093df03803f443"}
16:35:13 [16:35:13.426]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.428]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:903fd9df1f335530 epochNumber=1
{"provingJobId":"1:BASE_PARITY:903fd9df1f335530"}
16:35:13 [16:35:13.431]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKACMwM/1 with map size 1073741824
16:35:13 [16:35:13.434]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.577]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:903fd9df1f335530 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:903fd9df1f335530"}
16:35:13 [16:35:13.584]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7bd590644f29bc8 epochNumber=10
{"provingJobId":"1:BASE_PARITY:d7bd590644f29bc8"}
16:35:13 [16:35:13.585]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testKACMwM/10 with map size 1073741824
16:35:13 [16:35:13.587]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.630]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:13 [16:35:13.630]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:35:13 [16:35:13.737]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.738]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a9807a5611b4e3d epochNumber=1
{"provingJobId":"1:BASE_PARITY:6a9807a5611b4e3d"}
16:35:13 [16:35:13.740]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqFxyTL/1 with map size 1073741824
16:35:13 [16:35:13.743]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.916]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6a9807a5611b4e3d timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6a9807a5611b4e3d"}
16:35:13 [16:35:13.971]
INFO:
prover-client:proving-broker Proving Broker started
16:35:13 [16:35:13.972]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e7577b921374ebe epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e7577b921374ebe"}
16:35:13 [16:35:13.974]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkl3say/1 with map size 1073741824
16:35:13 [16:35:13.980]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:13 [16:35:13.996]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1e7577b921374ebe type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1e7577b921374ebe"}
16:35:14 [16:35:14.003]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.004]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:099c0e438728e518 epochNumber=1
{"provingJobId":"1:BASE_PARITY:099c0e438728e518"}
16:35:14 [16:35:14.005]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaRJhh5/1 with map size 1073741824
16:35:14 [16:35:14.007]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.025]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:099c0e438728e518 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:099c0e438728e518"}
16:35:14 [16:35:14.026]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:099c0e438728e518 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:099c0e438728e518"}
16:35:14 [16:35:14.035]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c45ca3185f17c6ce epochNumber=1
{"provingJobId":"1:BASE_PARITY:c45ca3185f17c6ce"}
16:35:14 [16:35:14.037]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQpiNsi/1 with map size 1073741824
16:35:14 [16:35:14.041]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.062]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c45ca3185f17c6ce type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c45ca3185f17c6ce"}
16:35:14 [16:35:14.073]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d63c905cbb854e7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:d63c905cbb854e7d"}
16:35:14 [16:35:14.075]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testM2udUU/1 with map size 1073741824
16:35:14 [16:35:14.081]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.104]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d63c905cbb854e7d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d63c905cbb854e7d"}
16:35:14 [16:35:14.105]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d185009780002254 epochNumber=3
{"provingJobId":"1:BASE_PARITY:d185009780002254"}
16:35:14 [16:35:14.107]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testM2udUU/3 with map size 1073741824
16:35:14 [16:35:14.109]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.149]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:14 [16:35:14.150]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:35:14 [16:35:14.154]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d63c905cbb854e7d err=test error
{"provingJoId":"1:BASE_PARITY:d63c905cbb854e7d"}
16:35:14 [16:35:14.171]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFcHLKo/1 with map size 1073741824
16:35:14 [16:35:14.177]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.197]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFcHLKo/2 with map size 1073741824
16:35:14 [16:35:14.200]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.225]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.228]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3bb6a449f3e3f4dc settled=false
{"provingJobId":"1:BASE_PARITY:3bb6a449f3e3f4dc","status":"pending"}
16:35:14 [16:35:14.229]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8222c13c605fe26b settled=false
{"provingJobId":"1:BASE_PARITY:8222c13c605fe26b","status":"pending"}
16:35:14 [16:35:14.238]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjSwXFp/1 with map size 1073741824
16:35:14 [16:35:14.241]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.266]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjSwXFp/2 with map size 1073741824
16:35:14 [16:35:14.271]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.308]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.314]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b8433cfee47c4815 settled=true
{"provingJobId":"1:BASE_PARITY:b8433cfee47c4815","status":"fulfilled"}
16:35:14 [16:35:14.318]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:a52fcb988e7ec7b6 settled=true
{"provingJobId":"2:BASE_PARITY:a52fcb988e7ec7b6","status":"fulfilled"}
16:35:14 [16:35:14.329]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZKRMYj/1 with map size 1073741824
16:35:14 [16:35:14.332]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.361]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZKRMYj/2 with map size 1073741824
16:35:14 [16:35:14.364]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.386]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.389]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d2d1788af54df4e1 settled=true
{"provingJobId":"1:BASE_PARITY:d2d1788af54df4e1","status":"fulfilled"}
16:35:14 [16:35:14.391]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6fbf582d8d9ef616 settled=false
{"provingJobId":"1:BASE_PARITY:6fbf582d8d9ef616","status":"pending"}
16:35:14 [16:35:14.401]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.402]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:538b82574d533afa epochNumber=1
{"provingJobId":"1:BASE_PARITY:538b82574d533afa"}
16:35:14 [16:35:14.413]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testILKweY/1 with map size 1073741824
16:35:14 [16:35:14.418]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.453]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.454]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dffedbcef6ca9106 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dffedbcef6ca9106"}
16:35:14 [16:35:14.455]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:dffedbcef6ca9106: Error: db error: Error: db error
16:35:14 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:dffedbcef6ca9106"}
16:35:14 [16:35:14.459]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d75bbee3b622e4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d75bbee3b622e4b"}
16:35:14 [16:35:14.460]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyO2LkV/1 with map size 1073741824
16:35:14 [16:35:14.465]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.483]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6d75bbee3b622e4b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6d75bbee3b622e4b"}
16:35:14 [16:35:14.484]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6d75bbee3b622e4b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6d75bbee3b622e4b"}
16:35:14 [16:35:14.498]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.499]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:082ce1e40ea3ff91 epochNumber=1
{"provingJobId":"1:BASE_PARITY:082ce1e40ea3ff91"}
16:35:14 [16:35:14.502]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQYiREz/1 with map size 1073741824
16:35:14 [16:35:14.515]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.536]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:082ce1e40ea3ff91 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:082ce1e40ea3ff91"}
16:35:14 [16:35:14.536]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:082ce1e40ea3ff91 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:082ce1e40ea3ff91"}
16:35:14 [16:35:14.537]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:082ce1e40ea3ff91: Error: db error
16:35:14 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:082ce1e40ea3ff91"}
16:35:14 [16:35:14.544]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1aa75d0b0780c12f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1aa75d0b0780c12f"}
16:35:14 [16:35:14.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNyrVHr/1 with map size 1073741824
16:35:14 [16:35:14.548]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.570]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1aa75d0b0780c12f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1aa75d0b0780c12f"}
16:35:14 [16:35:14.571]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1aa75d0b0780c12f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1aa75d0b0780c12f"}
16:35:14 [16:35:14.583]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.583]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15efe2aa1a2d6045 epochNumber=1
{"provingJobId":"1:BASE_PARITY:15efe2aa1a2d6045"}
16:35:14 [16:35:14.584]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSNom0U/1 with map size 1073741824
16:35:14 [16:35:14.588]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.611]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:15efe2aa1a2d6045 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:15efe2aa1a2d6045"}
16:35:14 [16:35:14.612]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:15efe2aa1a2d6045 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:15efe2aa1a2d6045"}
16:35:14 [16:35:14.613]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:15efe2aa1a2d6045 jobErr=test error: Error: db error
16:35:14 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:15efe2aa1a2d6045"}
16:35:14 [16:35:14.619]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.619]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d7640bd79a209024 not found
{"provingJobId":"1:BASE_PARITY:d7640bd79a209024"}
16:35:14 [16:35:14.628]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.628]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:396170d30453a7fd err=test error
{"provingJoId":"1:BASE_PARITY:396170d30453a7fd"}
16:35:14 [16:35:14.633]
INFO:
prover-client:proving-broker Proving Broker started
16:35:14 [16:35:14.644]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8261584e9166f5a epochNumber=1
{"provingJobId":"1:BASE_PARITY:f8261584e9166f5a"}
16:35:14 [16:35:14.645]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5gtHO5/1 with map size 1073741824
16:35:14 [16:35:14.648]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.671]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f8261584e9166f5a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f8261584e9166f5a"}
16:35:14 [16:35:14.672]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f8261584e9166f5a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f8261584e9166f5a"}
16:35:14 [16:35:14.690]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:7cbb51ffe3259a7e epochNumber=2
{"provingJobId":"2:BASE_PARITY:7cbb51ffe3259a7e"}
16:35:14 [16:35:14.691]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5gtHO5/2 with map size 1073741824
16:35:14 [16:35:14.695]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.719]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:7cbb51ffe3259a7e type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:7cbb51ffe3259a7e"}
16:35:14 [16:35:14.719]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:7cbb51ffe3259a7e type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:7cbb51ffe3259a7e"}
16:35:14 [16:35:14.787]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:8cae6bc83a202716 epochNumber=3
{"provingJobId":"3:BASE_PARITY:8cae6bc83a202716"}
16:35:14 [16:35:14.790]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test5gtHO5/3 with map size 1073741824
16:35:14 [16:35:14.794]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.838]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:14 [16:35:14.839]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:35:14 [16:35:14.877]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:593ddf517b0473ea epochNumber=4
{"provingJobId":"4:BASE_PARITY:593ddf517b0473ea"}
16:35:14 [16:35:14.880]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-test5gtHO5/4 with map size 1073741824
16:35:14 [16:35:14.887]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:14 [16:35:14.921]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:14 [16:35:14.921]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
16:35:14 [16:35:14.972]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:20c39620a23e1947 epochNumber=5
{"provingJobId":"5:BASE_PARITY:20c39620a23e1947"}
16:35:14 [16:35:14.974]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-test5gtHO5/5 with map size 1073741824
16:35:14 [16:35:14.977]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:35:15 [16:35:15.000]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:35:15 [16:35:15.001]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
16:35:15
PASS src/proving_broker/proving_broker.test.ts (
13.409 s)
16:35:15 ProvingBroker
16:35:15 Producer API
16:35:15
✓ refuses stale jobs (53 ms)
16:35:15
✓ enqueues jobs (5 ms)
16:35:15
✓ ignores duplicate jobs (5 ms)
16:35:15
✓ reports correct status when enqueuing repeat jobs (5 ms)
16:35:15
✓ throws an error in case of duplicate job IDs (30 ms)
16:35:15
✓ returns not-found status for non-existing jobs (2 ms)
16:35:15
✓ cancels jobs in queue (6 ms)
16:35:15
✓ cancels jobs in-progress (3 ms)
16:35:15
✓ returns job result if successful (4 ms)
16:35:15
✓ returns job error if failed (3 ms)
16:35:15
✓ correctly returns job status for concurrent writes (3 ms)
16:35:15
✓ refuses stale jobs (45 ms)
16:35:15
✓ enqueues jobs (37 ms)
16:35:15
✓ ignores duplicate jobs (32 ms)
16:35:15
✓ reports correct status when enqueuing repeat jobs (35 ms)
16:35:15
✓ throws an error in case of duplicate job IDs (28 ms)
16:35:15
✓ returns not-found status for non-existing jobs (4 ms)
16:35:15
✓ cancels jobs in queue (33 ms)
16:35:15
✓ cancels jobs in-progress (29 ms)
16:35:15
✓ returns job result if successful (35 ms)
16:35:15
✓ returns job error if failed (42 ms)
16:35:15
✓ correctly returns job status for concurrent writes (39 ms)
16:35:15 Consumer API
16:35:15
✓ returns undefined if no jobs are available (10 ms)
16:35:15
✓ returns jobs in priority order (2 ms)
16:35:15
✓ returns undefined if no jobs are available for the given allowList (1 ms)
16:35:15
✓ returns a job if it is in the allowList (3 ms)
16:35:15
✓ returns the most important job if it is in the allowList (2 ms)
16:35:15
✓ returns any job if filter is empty (2 ms)
16:35:15
✓ returns a new job when reporting job success (2 ms)
16:35:15
✓ returns a new job when reporting permanent error (2 ms)
16:35:15
✓ returns a new job when reporting retry-able error (2 ms)
16:35:15
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
16:35:15
✓ returns a new job if job is already in progress elsewhere (1057 ms)
16:35:15
✓ avoids sending the same job to a new agent after a restart (1007 ms)
16:35:15
✓ avoids sending a completed job to a new agent after a restart (1004 ms)
16:35:15
✓ tracks job result if in progress (3 ms)
16:35:15
✓ tracks job result even if job is in queue (2 ms)
16:35:15
✓ ignores reported job error if unknown job (2 ms)
16:35:15
✓ ignores job result if unknown job (1 ms)
16:35:15
✓ returns undefined if no jobs are available (2 ms)
16:35:15
✓ returns jobs in priority order (52 ms)
16:35:15
✓ returns undefined if no jobs are available for the given allowList (23 ms)
16:35:15
✓ returns a job if it is in the allowList (58 ms)
16:35:15
✓ returns the most important job if it is in the allowList (55 ms)
16:35:15
✓ returns any job if filter is empty (55 ms)
16:35:15
✓ returns a new job when reporting job success (41 ms)
16:35:15
✓ returns a new job when reporting permanent error (39 ms)
16:35:15
✓ returns a new job when reporting retry-able error (32 ms)
16:35:15
✓ returns a new job when reporting progress if current one is cancelled (33 ms)
16:35:15
✓ returns a new job if job is already in progress elsewhere (1119 ms)
16:35:15
✓ avoids sending the same job to a new agent after a restart (1058 ms)
16:35:15
✓ avoids sending a completed job to a new agent after a restart (1066 ms)
16:35:15
✓ tracks job result if in progress (59 ms)
16:35:15
✓ tracks job result even if job is in queue (63 ms)
16:35:15
✓ ignores reported job error if unknown job (5 ms)
16:35:15
✓ ignores job result if unknown job (5 ms)
16:35:15 Timeouts
16:35:15
✓ tracks in progress jobs (1 ms)
16:35:15
✓ re-enqueues jobs that time out (109 ms)
16:35:15
✓ cancel stale jobs that time out (230 ms)
16:35:15
✓ keeps the jobs in progress while it is alive (204 ms)
16:35:15
✓ tracks in progress jobs (30 ms)
16:35:15
✓ re-enqueues jobs that time out (140 ms)
16:35:15
✓ cancel stale jobs that time out (311 ms)
16:35:15
✓ keeps the jobs in progress while it is alive (234 ms)
16:35:15 Retries
16:35:15
✓ retries jobs (2 ms)
16:35:15
✓ retries up to a maximum number of times (3 ms)
16:35:15
✓ passing retry=false does not retry (1 ms)
16:35:15
✓ does not retry if job is stale (27 ms)
16:35:15
✓ retries jobs (31 ms)
16:35:15
✓ retries up to a maximum number of times (32 ms)
16:35:15
✓ passing retry=false does not retry (37 ms)
16:35:15
✓ does not retry if job is stale (86 ms)
16:35:15 Database management
16:35:15
✓ re-enqueues proof requests on start (3 ms)
16:35:15
✓ restores proof results on start (2 ms)
16:35:15
✓ only re-enqueues unfinished jobs (1 ms)
16:35:15
✓ saves job when enqueued (3 ms)
16:35:15
✓ does not retain job if database fails to save (2 ms)
16:35:15
✓ saves job result (1 ms)
16:35:15
✓ saves result even if database fails to save (2 ms)
16:35:15
✓ saves job error (2 ms)
16:35:15
✓ saves job error even if database fails to save (3 ms)
16:35:15
✓ does not save job result if job is unknown (1 ms)
16:35:15
✓ does not save job error if job is unknown (1 ms)
16:35:15
✓ cleans up old jobs periodically (264 ms)
16:35:15
✓ re-enqueues proof requests on start (75 ms)
16:35:15
✓ restores proof results on start (88 ms)
16:35:15
✓ only re-enqueues unfinished jobs (72 ms)
16:35:15
✓ saves job when enqueued (53 ms)
16:35:15
✓ does not retain job if database fails to save (8 ms)
16:35:15
✓ saves job result (35 ms)
16:35:15
✓ saves result even if database fails to save (46 ms)
16:35:15
✓ saves job error (40 ms)
16:35:15
✓ saves job error even if database fails to save (35 ms)
16:35:15
✓ does not save job result if job is unknown (5 ms)
16:35:15
✓ does not save job error if job is unknown (8 ms)
16:35:15
✓ cleans up old jobs periodically (421 ms)
16:35:15
16:35:15
Test Suites: 1 passed, 1 total
16:35:15
Tests: 96 passed, 96 total
16:35:15
Snapshots: 0 total
16:35:15
Time: 13.508 s
16:35:15
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
16:35:15
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?