Parent Log:
http://ci.aztec-labs.com/94393e9a39f896cc
Command: 2f5a01bdc943190c yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/b4e88ec5c5067a9b12be306fb60258508bcfdcf8
Env: REF_NAME=gh-readonly-queue/next/pr-15196-812a53d5d7c2057744dbf808172864443b5de7af CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Thu Jun 26 14:23:46 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15196_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
14:23:50 [14:23:50.953]
INFO:
prover-client:proving-broker Proving Broker started
14:23:50 [14:23:50.958]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97018c3c8fe9438d epochNumber=42
{"provingJobId":"1:BASE_PARITY:97018c3c8fe9438d"}
14:23:50 [14:23:50.961]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:9013999d01ae1552 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9013999d01ae1552"}
14:23:50 [14:23:50.995]
INFO:
prover-client:proving-broker Proving Broker started
14:23:50 [14:23:50.995]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e6ad67396d84c93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2e6ad67396d84c93"}
14:23:50 [14:23:50.997]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df434e46707b2a98 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df434e46707b2a98"}
14:23:50 [14:23:50.998]
INFO:
prover-client:proving-broker Proving Broker started
14:23:50 [14:23:50.999]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d309a1d7ec345ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:2d309a1d7ec345ac"}
14:23:51 [14:23:51.000]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2d309a1d7ec345ac epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2d309a1d7ec345ac"}
14:23:51 [14:23:51.002]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.003]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c6e39e9038c5fdc epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c6e39e9038c5fdc"}
14:23:51 [14:23:51.004]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6c6e39e9038c5fdc epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6c6e39e9038c5fdc"}
14:23:51 [14:23:51.004]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6c6e39e9038c5fdc type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6c6e39e9038c5fdc"}
14:23:51 [14:23:51.005]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6c6e39e9038c5fdc epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6c6e39e9038c5fdc"}
14:23:51 [14:23:51.006]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.006]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd3b281266bacdcb epochNumber=1
{"provingJobId":"1:BASE_PARITY:cd3b281266bacdcb"}
14:23:51 [14:23:51.031]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.033]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:840a01770f438ccb epochNumber=1
{"provingJobId":"1:BASE_PARITY:840a01770f438ccb"}
14:23:51 [14:23:51.034]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:840a01770f438ccb
{"provingJobId":"1:BASE_PARITY:840a01770f438ccb"}
14:23:51 [14:23:51.035]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:840a01770f438ccb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:840a01770f438ccb"}
14:23:51 [14:23:51.035]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:840a01770f438ccb type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:840a01770f438ccb"}
14:23:51 [14:23:51.036]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.037]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:278540eb82485aae epochNumber=1
{"provingJobId":"1:BASE_PARITY:278540eb82485aae"}
14:23:51 [14:23:51.037]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:278540eb82485aae
{"provingJobId":"1:BASE_PARITY:278540eb82485aae"}
14:23:51 [14:23:51.038]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:278540eb82485aae type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:278540eb82485aae"}
14:23:51 [14:23:51.039]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.040]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9dab841d4ef202eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:9dab841d4ef202eb"}
14:23:51 [14:23:51.040]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9dab841d4ef202eb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9dab841d4ef202eb"}
14:23:51 [14:23:51.040]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9dab841d4ef202eb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9dab841d4ef202eb"}
14:23:51 [14:23:51.042]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.043]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff9740b4115f4f7c epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff9740b4115f4f7c"}
14:23:51 [14:23:51.043]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ff9740b4115f4f7c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ff9740b4115f4f7c"}
14:23:51 [14:23:51.043]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ff9740b4115f4f7c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ff9740b4115f4f7c"}
14:23:51 [14:23:51.044]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.045]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f3f2828edb36b41 epochNumber=0
{"provingJobId":"1:BASE_PARITY:7f3f2828edb36b41"}
14:23:51 [14:23:51.045]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7f3f2828edb36b41 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7f3f2828edb36b41"}
14:23:51 [14:23:51.046]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ffa0e310a50180dd epochNumber=0
{"provingJobId":"1:BASE_PARITY:ffa0e310a50180dd"}
14:23:51 [14:23:51.046]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7f3f2828edb36b41 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7f3f2828edb36b41"}
14:23:51 [14:23:51.046]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83c9373624aebc03 epochNumber=0
{"provingJobId":"1:BASE_PARITY:83c9373624aebc03"}
14:23:51 [14:23:51.047]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.049]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.049]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d626d3b37091a2f6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d626d3b37091a2f6"}
14:23:51 [14:23:51.049]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:292cf89a29f37621 epochNumber=1
{"provingJobId":"1:BASE_PARITY:292cf89a29f37621"}
14:23:51 [14:23:51.051]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.051]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a2d1a5a4074c27a epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a2d1a5a4074c27a"}
14:23:51 [14:23:51.052]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.053]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df3bfe229c36a63a epochNumber=1
{"provingJobId":"1:BASE_PARITY:df3bfe229c36a63a"}
14:23:51 [14:23:51.053]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d800b18c3d7c407 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d800b18c3d7c407"}
14:23:51 [14:23:51.053]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cde807171f97a55f epochNumber=2
{"provingJobId":"1:BASE_PARITY:cde807171f97a55f"}
14:23:51 [14:23:51.054]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc835fa4deb88f2c epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc835fa4deb88f2c"}
14:23:51 [14:23:51.061]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7567fdd15ea31fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7567fdd15ea31fe"}
14:23:51 [14:23:51.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:265b1a4eed367442 epochNumber=1
{"provingJobId":"1:BASE_PARITY:265b1a4eed367442"}
14:23:51 [14:23:51.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08985cf30d409058 epochNumber=2
{"provingJobId":"1:BASE_PARITY:08985cf30d409058"}
14:23:51 [14:23:51.064]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f022b0fd43faeede epochNumber=1
{"provingJobId":"1:BASE_PARITY:f022b0fd43faeede"}
14:23:51 [14:23:51.065]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.066]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3138a1a400bc4bfd epochNumber=1
{"provingJobId":"1:BASE_PARITY:3138a1a400bc4bfd"}
14:23:51 [14:23:51.066]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d21072e9a6a0586a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d21072e9a6a0586a"}
14:23:51 [14:23:51.067]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d2a950a24e60979 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2d2a950a24e60979"}
14:23:51 [14:23:51.067]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e67c07f037ef0eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e67c07f037ef0eb"}
14:23:51 [14:23:51.068]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.069]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22a18cf51b10d07c epochNumber=1
{"provingJobId":"1:BASE_PARITY:22a18cf51b10d07c"}
14:23:51 [14:23:51.069]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36d4c9e06f7fd02e epochNumber=1
{"provingJobId":"1:BASE_PARITY:36d4c9e06f7fd02e"}
14:23:51 [14:23:51.069]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:22a18cf51b10d07c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:22a18cf51b10d07c"}
14:23:51 [14:23:51.071]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.071]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:201d897ade263179 epochNumber=1
{"provingJobId":"1:BASE_PARITY:201d897ade263179"}
14:23:51 [14:23:51.072]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36af287732cede70 epochNumber=1
{"provingJobId":"1:BASE_PARITY:36af287732cede70"}
14:23:51 [14:23:51.072]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:201d897ade263179 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:201d897ade263179"}
14:23:51 [14:23:51.073]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56d497553819d4c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56d497553819d4c9"}
14:23:51 [14:23:51.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ae672f0477a966a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ae672f0477a966a"}
14:23:51 [14:23:51.074]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:56d497553819d4c9 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:56d497553819d4c9"}
14:23:51 [14:23:51.075]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.076]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2fd10ad86b7a0927 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2fd10ad86b7a0927"}
14:23:51 [14:23:51.076]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2fd10ad86b7a0927
{"provingJobId":"1:BASE_PARITY:2fd10ad86b7a0927"}
14:23:51 [14:23:51.076]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2fd10ad86b7a0927 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2fd10ad86b7a0927"}
14:23:51 [14:23:51.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a841ab8237722d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a841ab8237722d9"}
14:23:51 [14:23:51.077]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2fd10ad86b7a0927 has already been completed
{"provingJobId":"1:BASE_PARITY:2fd10ad86b7a0927"}
14:23:51 [14:23:51.078]
INFO:
prover-client:proving-broker Proving Broker started
14:23:51 [14:23:51.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd497465592f1d03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd497465592f1d03"}
14:23:51 [14:23:51.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c37aa7b025b0f1d epochNumber=2
{"provingJobId":"1:BASE_PARITY:3c37aa7b025b0f1d"}
14:23:52 [14:23:52.133]
INFO:
prover-client:proving-broker Proving Broker started
14:23:52 [14:23:52.135]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fd497465592f1d03 settled=false
{"provingJobId":"1:BASE_PARITY:fd497465592f1d03","status":"pending"}
14:23:52 [14:23:52.135]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3c37aa7b025b0f1d settled=false
{"provingJobId":"1:BASE_PARITY:3c37aa7b025b0f1d","status":"pending"}
14:23:52 [14:23:52.137]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd497465592f1d03 type=BASE_PARITY startedAt=1750947831080 older agent has taken job
{"provingJobId":"1:BASE_PARITY:fd497465592f1d03"}
14:23:52 [14:23:52.137]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd497465592f1d03 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:fd497465592f1d03"}
14:23:52 [14:23:52.139]
INFO:
prover-client:proving-broker Proving Broker started
14:23:52 [14:23:52.141]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:844c57546d3c27e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:844c57546d3c27e2"}
14:23:52 [14:23:52.141]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54e800e36ec884c7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:54e800e36ec884c7"}
14:23:53 [14:23:53.143]
INFO:
prover-client:proving-broker Proving Broker started
14:23:53 [14:23:53.143]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:844c57546d3c27e2 settled=false
{"provingJobId":"1:BASE_PARITY:844c57546d3c27e2","status":"pending"}
14:23:53 [14:23:53.144]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:54e800e36ec884c7 settled=false
{"provingJobId":"1:BASE_PARITY:54e800e36ec884c7","status":"pending"}
14:23:53 [14:23:53.145]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:844c57546d3c27e2 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:844c57546d3c27e2"}
14:23:53 [14:23:53.148]
INFO:
prover-client:proving-broker Proving Broker started
14:23:53 [14:23:53.149]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca38a71b048de08a epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca38a71b048de08a"}
14:23:53 [14:23:53.149]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f868115a29a466ef epochNumber=2
{"provingJobId":"1:BASE_PARITY:f868115a29a466ef"}
14:23:54 [14:23:54.151]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.151]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ca38a71b048de08a settled=false
{"provingJobId":"1:BASE_PARITY:ca38a71b048de08a","status":"pending"}
14:23:54 [14:23:54.152]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f868115a29a466ef settled=false
{"provingJobId":"1:BASE_PARITY:f868115a29a466ef","status":"pending"}
14:23:54 [14:23:54.153]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ca38a71b048de08a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ca38a71b048de08a"}
14:23:54 [14:23:54.153]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ca38a71b048de08a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ca38a71b048de08a"}
14:23:54 [14:23:54.155]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.156]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53b1b02b4e84b0a2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53b1b02b4e84b0a2"}
14:23:54 [14:23:54.157]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c493b34d16f44daa epochNumber=2
{"provingJobId":"1:BASE_PARITY:c493b34d16f44daa"}
14:23:54 [14:23:54.158]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:53b1b02b4e84b0a2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:53b1b02b4e84b0a2"}
14:23:54 [14:23:54.158]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c493b34d16f44daa type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c493b34d16f44daa"}
14:23:54 [14:23:54.160]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.160]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eefc513d52434dde epochNumber=1
{"provingJobId":"1:BASE_PARITY:eefc513d52434dde"}
14:23:54 [14:23:54.161]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:234042082ee5e683 epochNumber=2
{"provingJobId":"1:BASE_PARITY:234042082ee5e683"}
14:23:54 [14:23:54.161]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eefc513d52434dde type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:eefc513d52434dde"}
14:23:54 [14:23:54.161]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:eefc513d52434dde type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:eefc513d52434dde"}
14:23:54 [14:23:54.162]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:234042082ee5e683 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:234042082ee5e683"}
14:23:54 [14:23:54.163]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.164]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bb6c79da1b962a6a err=test error
{"provingJoId":"1:BASE_PARITY:bb6c79da1b962a6a"}
14:23:54 [14:23:54.165]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.166]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cc1582daf41609f4 not found
{"provingJobId":"1:BASE_PARITY:cc1582daf41609f4"}
14:23:54 [14:23:54.168]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.168]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d021a7cc833d58b epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d021a7cc833d58b"}
14:23:54 [14:23:54.170]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.171]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1298ece7cf86fdc epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1298ece7cf86fdc"}
14:23:54 [14:23:54.273]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d1298ece7cf86fdc timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:d1298ece7cf86fdc"}
14:23:54 [14:23:54.280]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.280]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa1b4073ae59fa41 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa1b4073ae59fa41"}
14:23:54 [14:23:54.382]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fa1b4073ae59fa41 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:fa1b4073ae59fa41"}
14:23:54 [14:23:54.389]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d52f6c57ee8dbfa5 epochNumber=10
{"provingJobId":"1:BASE_PARITY:d52f6c57ee8dbfa5"}
14:23:54 [14:23:54.408]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:23:54 [14:23:54.517]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.518]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d05c33d7bc312da epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d05c33d7bc312da"}
14:23:54 [14:23:54.671]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3d05c33d7bc312da timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:3d05c33d7bc312da"}
14:23:54 [14:23:54.722]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.723]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1b83a0f58453458 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1b83a0f58453458"}
14:23:54 [14:23:54.723]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d1b83a0f58453458 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d1b83a0f58453458"}
14:23:54 [14:23:54.724]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.725]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1801dc29e355db66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1801dc29e355db66"}
14:23:54 [14:23:54.726]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1801dc29e355db66 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1801dc29e355db66"}
14:23:54 [14:23:54.726]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1801dc29e355db66 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:1801dc29e355db66"}
14:23:54 [14:23:54.727]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.728]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56a0220968a49c7e epochNumber=1
{"provingJobId":"1:BASE_PARITY:56a0220968a49c7e"}
14:23:54 [14:23:54.728]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:56a0220968a49c7e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:56a0220968a49c7e"}
14:23:54 [14:23:54.729]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.730]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f8f2e7b62e2390a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f8f2e7b62e2390a"}
14:23:54 [14:23:54.730]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6f8f2e7b62e2390a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6f8f2e7b62e2390a"}
14:23:54 [14:23:54.731]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2c35dedb54494cf epochNumber=3
{"provingJobId":"1:BASE_PARITY:b2c35dedb54494cf"}
14:23:54 [14:23:54.755]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:23:54 [14:23:54.755]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6f8f2e7b62e2390a err=test error
{"provingJoId":"1:BASE_PARITY:6f8f2e7b62e2390a"}
14:23:54 [14:23:54.757]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.758]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e097905ebfb81909 settled=false
{"provingJobId":"1:BASE_PARITY:e097905ebfb81909","status":"pending"}
14:23:54 [14:23:54.758]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:509e3855ac415fc9 settled=false
{"provingJobId":"1:BASE_PARITY:509e3855ac415fc9","status":"pending"}
14:23:54 [14:23:54.760]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.761]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fcbfbe61064cf5ae settled=true
{"provingJobId":"1:BASE_PARITY:fcbfbe61064cf5ae","status":"fulfilled"}
14:23:54 [14:23:54.761]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:bebf3c9080580050 settled=true
{"provingJobId":"2:BASE_PARITY:bebf3c9080580050","status":"fulfilled"}
14:23:54 [14:23:54.762]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.763]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2cb415d79a402841 settled=true
{"provingJobId":"1:BASE_PARITY:2cb415d79a402841","status":"fulfilled"}
14:23:54 [14:23:54.763]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e61db004d889d5f7 settled=false
{"provingJobId":"1:BASE_PARITY:e61db004d889d5f7","status":"pending"}
14:23:54 [14:23:54.764]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.766]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51025325ab948d20 epochNumber=1
{"provingJobId":"1:BASE_PARITY:51025325ab948d20"}
14:23:54 [14:23:54.768]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d60126f160769403 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d60126f160769403"}
14:23:54 [14:23:54.769]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:d60126f160769403: Error: db error: Error: db error
14:23:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:d60126f160769403"}
14:23:54 [14:23:54.771]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.771]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f41e5f8376b9646 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f41e5f8376b9646"}
14:23:54 [14:23:54.771]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3f41e5f8376b9646 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3f41e5f8376b9646"}
14:23:54 [14:23:54.771]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3f41e5f8376b9646 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3f41e5f8376b9646"}
14:23:54 [14:23:54.773]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.773]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:258e90d3cb838936 epochNumber=1
{"provingJobId":"1:BASE_PARITY:258e90d3cb838936"}
14:23:54 [14:23:54.773]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:258e90d3cb838936 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:258e90d3cb838936"}
14:23:54 [14:23:54.773]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:258e90d3cb838936 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:258e90d3cb838936"}
14:23:54 [14:23:54.774]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:258e90d3cb838936: Error: db error
14:23:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:258e90d3cb838936"}
14:23:54 [14:23:54.775]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c936dc234c9af793 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c936dc234c9af793"}
14:23:54 [14:23:54.776]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c936dc234c9af793 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c936dc234c9af793"}
14:23:54 [14:23:54.776]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c936dc234c9af793 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c936dc234c9af793"}
14:23:54 [14:23:54.777]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a36b5b49092547f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a36b5b49092547f8"}
14:23:54 [14:23:54.777]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a36b5b49092547f8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a36b5b49092547f8"}
14:23:54 [14:23:54.778]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a36b5b49092547f8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a36b5b49092547f8"}
14:23:54 [14:23:54.778]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a36b5b49092547f8 jobErr=test error: Error: db error
14:23:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:a36b5b49092547f8"}
14:23:54 [14:23:54.779]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.780]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:31957550db559d15 not found
{"provingJobId":"1:BASE_PARITY:31957550db559d15"}
14:23:54 [14:23:54.781]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.781]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e28ad32f36e0d39d err=test error
{"provingJoId":"1:BASE_PARITY:e28ad32f36e0d39d"}
14:23:54 [14:23:54.787]
INFO:
prover-client:proving-broker Proving Broker started
14:23:54 [14:23:54.797]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef2ce3ede4770e72 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef2ce3ede4770e72"}
14:23:54 [14:23:54.798]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ef2ce3ede4770e72 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ef2ce3ede4770e72"}
14:23:54 [14:23:54.798]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ef2ce3ede4770e72 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ef2ce3ede4770e72"}
14:23:54 [14:23:54.809]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:015ca93975609f94 epochNumber=2
{"provingJobId":"2:BASE_PARITY:015ca93975609f94"}
14:23:54 [14:23:54.809]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:015ca93975609f94 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:015ca93975609f94"}
14:23:54 [14:23:54.810]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:015ca93975609f94 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:015ca93975609f94"}
14:23:54 [14:23:54.871]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:dd2cd46ad7e1d0e4 epochNumber=3
{"provingJobId":"3:BASE_PARITY:dd2cd46ad7e1d0e4"}
14:23:54 [14:23:54.888]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:23:54 [14:23:54.935]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:4fd55b9dbbdfa97c epochNumber=4
{"provingJobId":"4:BASE_PARITY:4fd55b9dbbdfa97c"}
14:23:54 [14:23:54.941]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:23:54 [14:23:54.997]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:69f8c479109eba55 epochNumber=5
{"provingJobId":"5:BASE_PARITY:69f8c479109eba55"}
14:23:55 [14:23:55.017]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:23:55 [14:23:55.052]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.053]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d83aea4d7134bf5c epochNumber=42
{"provingJobId":"1:BASE_PARITY:d83aea4d7134bf5c"}
14:23:55 [14:23:55.054]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-test9CrGQ0/42 with map size 1073741824
14:23:55 [14:23:55.058]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.082]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:bf88105960c98eba epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf88105960c98eba"}
14:23:55 [14:23:55.086]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f21a03a231a1e26c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f21a03a231a1e26c"}
14:23:55 [14:23:55.087]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testI7RqUd/1 with map size 1073741824
14:23:55 [14:23:55.089]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.107]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e05d15d25c0d5f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e05d15d25c0d5f0"}
14:23:55 [14:23:55.118]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.120]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:adf74cc5df9143c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:adf74cc5df9143c2"}
14:23:55 [14:23:55.121]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testombPX1/1 with map size 1073741824
14:23:55 [14:23:55.123]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.140]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:adf74cc5df9143c2 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:adf74cc5df9143c2"}
14:23:55 [14:23:55.146]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.147]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92ff5609fd8543c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:92ff5609fd8543c8"}
14:23:55 [14:23:55.148]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBw7xSt/1 with map size 1073741824
14:23:55 [14:23:55.149]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.166]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:92ff5609fd8543c8 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:92ff5609fd8543c8"}
14:23:55 [14:23:55.167]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:92ff5609fd8543c8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:92ff5609fd8543c8"}
14:23:55 [14:23:55.174]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:92ff5609fd8543c8 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:92ff5609fd8543c8"}
14:23:55 [14:23:55.176]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.177]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b51ac4a60d779263 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b51ac4a60d779263"}
14:23:55 [14:23:55.177]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testldDxet/1 with map size 1073741824
14:23:55 [14:23:55.179]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.199]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.202]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.203]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89b02995651a5265 epochNumber=1
{"provingJobId":"1:BASE_PARITY:89b02995651a5265"}
14:23:55 [14:23:55.203]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testB3mAbv/1 with map size 1073741824
14:23:55 [14:23:55.207]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.223]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:89b02995651a5265
{"provingJobId":"1:BASE_PARITY:89b02995651a5265"}
14:23:55 [14:23:55.224]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:89b02995651a5265 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:89b02995651a5265"}
14:23:55 [14:23:55.224]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:89b02995651a5265 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:89b02995651a5265"}
14:23:55 [14:23:55.233]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.234]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c250abe4d04b34d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c250abe4d04b34d9"}
14:23:55 [14:23:55.235]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testby4nIs/1 with map size 1073741824
14:23:55 [14:23:55.237]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.257]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c250abe4d04b34d9
{"provingJobId":"1:BASE_PARITY:c250abe4d04b34d9"}
14:23:55 [14:23:55.257]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c250abe4d04b34d9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c250abe4d04b34d9"}
14:23:55 [14:23:55.268]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7151375ecdaf58d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7151375ecdaf58d7"}
14:23:55 [14:23:55.269]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwxq9QE/1 with map size 1073741824
14:23:55 [14:23:55.270]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.288]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7151375ecdaf58d7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7151375ecdaf58d7"}
14:23:55 [14:23:55.289]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7151375ecdaf58d7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7151375ecdaf58d7"}
14:23:55 [14:23:55.297]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.298]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:556bb630d859c7e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:556bb630d859c7e3"}
14:23:55 [14:23:55.298]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbw7KPY/1 with map size 1073741824
14:23:55 [14:23:55.301]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.318]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:556bb630d859c7e3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:556bb630d859c7e3"}
14:23:55 [14:23:55.319]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:556bb630d859c7e3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:556bb630d859c7e3"}
14:23:55 [14:23:55.327]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.327]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8a69bd7447f0c8c epochNumber=0
{"provingJobId":"1:BASE_PARITY:e8a69bd7447f0c8c"}
14:23:55 [14:23:55.328]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testNyHBGC/0 with map size 1073741824
14:23:55 [14:23:55.329]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.345]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e8a69bd7447f0c8c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e8a69bd7447f0c8c"}
14:23:55 [14:23:55.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb2bb78a3374a19b epochNumber=0
{"provingJobId":"1:BASE_PARITY:bb2bb78a3374a19b"}
14:23:55 [14:23:55.347]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e8a69bd7447f0c8c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e8a69bd7447f0c8c"}
14:23:55 [14:23:55.347]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d7dc2d4c8f0c8b2 epochNumber=0
{"provingJobId":"1:BASE_PARITY:3d7dc2d4c8f0c8b2"}
14:23:55 [14:23:55.360]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.363]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.363]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56bbbbcb6668d635 epochNumber=2
{"provingJobId":"1:BASE_PARITY:56bbbbcb6668d635"}
14:23:55 [14:23:55.364]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testoD4iue/2 with map size 1073741824
14:23:55 [14:23:55.366]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.383]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f53023e58daa65c epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f53023e58daa65c"}
14:23:55 [14:23:55.384]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoD4iue/1 with map size 1073741824
14:23:55 [14:23:55.387]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.405]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.406]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:260f35ec2d57a8dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:260f35ec2d57a8dd"}
14:23:55 [14:23:55.409]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3kX46p/1 with map size 1073741824
14:23:55 [14:23:55.411]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.429]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.430]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8426714bf5fadf75 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8426714bf5fadf75"}
14:23:55 [14:23:55.430]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHtZcAB/1 with map size 1073741824
14:23:55 [14:23:55.432]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88e15747b2f5f215 epochNumber=1
{"provingJobId":"1:BASE_PARITY:88e15747b2f5f215"}
14:23:55 [14:23:55.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:afa14004bb89bb1b epochNumber=2
{"provingJobId":"1:BASE_PARITY:afa14004bb89bb1b"}
14:23:55 [14:23:55.453]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testHtZcAB/2 with map size 1073741824
14:23:55 [14:23:55.455]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.472]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66f398430121091c epochNumber=1
{"provingJobId":"1:BASE_PARITY:66f398430121091c"}
14:23:55 [14:23:55.481]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7893383f451d81a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7893383f451d81a8"}
14:23:55 [14:23:55.482]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNbV6Xe/1 with map size 1073741824
14:23:55 [14:23:55.484]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.504]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bdc86228fdfa750f epochNumber=1
{"provingJobId":"1:BASE_PARITY:bdc86228fdfa750f"}
14:23:55 [14:23:55.511]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1a9682d2069aee4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a1a9682d2069aee4"}
14:23:55 [14:23:55.513]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNbV6Xe/2 with map size 1073741824
14:23:55 [14:23:55.514]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.533]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2dce201cb4f47c91 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2dce201cb4f47c91"}
14:23:55 [14:23:55.543]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50a467a47b4e4021 epochNumber=1
{"provingJobId":"1:BASE_PARITY:50a467a47b4e4021"}
14:23:55 [14:23:55.544]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWWo4T2/1 with map size 1073741824
14:23:55 [14:23:55.546]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.562]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df54073d5f2e1c81 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df54073d5f2e1c81"}
14:23:55 [14:23:55.568]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c61dc1eea29624d epochNumber=2
{"provingJobId":"1:BASE_PARITY:1c61dc1eea29624d"}
14:23:55 [14:23:55.569]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWWo4T2/2 with map size 1073741824
14:23:55 [14:23:55.570]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.589]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a3b45eef556cc46 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a3b45eef556cc46"}
14:23:55 [14:23:55.599]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.599]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0aca7b36dbd9225 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c0aca7b36dbd9225"}
14:23:55 [14:23:55.600]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7RupvK/1 with map size 1073741824
14:23:55 [14:23:55.602]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.624]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08f07674d95d366f epochNumber=1
{"provingJobId":"1:BASE_PARITY:08f07674d95d366f"}
14:23:55 [14:23:55.630]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c0aca7b36dbd9225 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c0aca7b36dbd9225"}
14:23:55 [14:23:55.640]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.640]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b67711866d87c53 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b67711866d87c53"}
14:23:55 [14:23:55.641]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUV8bfY/1 with map size 1073741824
14:23:55 [14:23:55.643]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.660]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98177a605cd729c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:98177a605cd729c1"}
14:23:55 [14:23:55.666]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9b67711866d87c53 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:9b67711866d87c53"}
14:23:55 [14:23:55.677]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.677]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0a0e6095eed42d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f0a0e6095eed42d0"}
14:23:55 [14:23:55.678]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxo5qAp/1 with map size 1073741824
14:23:55 [14:23:55.683]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.704]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae822d7d8bb1726e epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae822d7d8bb1726e"}
14:23:55 [14:23:55.711]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f0a0e6095eed42d0 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:f0a0e6095eed42d0"}
14:23:55 [14:23:55.716]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7440dbdcbcf4651e epochNumber=1
{"provingJobId":"1:BASE_PARITY:7440dbdcbcf4651e"}
14:23:55 [14:23:55.718]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testThnqLv/1 with map size 1073741824
14:23:55 [14:23:55.719]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.736]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7440dbdcbcf4651e
{"provingJobId":"1:BASE_PARITY:7440dbdcbcf4651e"}
14:23:55 [14:23:55.737]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7440dbdcbcf4651e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7440dbdcbcf4651e"}
14:23:55 [14:23:55.743]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f85dede13a16a19 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f85dede13a16a19"}
14:23:55 [14:23:55.750]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7440dbdcbcf4651e has already been completed
{"provingJobId":"1:BASE_PARITY:7440dbdcbcf4651e"}
14:23:55 [14:23:55.755]
INFO:
prover-client:proving-broker Proving Broker started
14:23:55 [14:23:55.756]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9cc5374636fbf01e epochNumber=1
{"provingJobId":"1:BASE_PARITY:9cc5374636fbf01e"}
14:23:55 [14:23:55.756]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJrDXrQ/1 with map size 1073741824
14:23:55 [14:23:55.758]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:55 [14:23:55.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:865bfced98078a6a epochNumber=2
{"provingJobId":"1:BASE_PARITY:865bfced98078a6a"}
14:23:55 [14:23:55.777]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJrDXrQ/2 with map size 1073741824
14:23:55 [14:23:55.780]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:56 [14:23:56.855]
INFO:
prover-client:proving-broker Proving Broker started
14:23:56 [14:23:56.861]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9cc5374636fbf01e settled=false
{"provingJobId":"1:BASE_PARITY:9cc5374636fbf01e","status":"pending"}
14:23:56 [14:23:56.863]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:865bfced98078a6a settled=false
{"provingJobId":"1:BASE_PARITY:865bfced98078a6a","status":"pending"}
14:23:56 [14:23:56.865]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9cc5374636fbf01e type=BASE_PARITY startedAt=1750947835802 older agent has taken job
{"provingJobId":"1:BASE_PARITY:9cc5374636fbf01e"}
14:23:56 [14:23:56.866]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9cc5374636fbf01e type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:9cc5374636fbf01e"}
14:23:56 [14:23:56.873]
INFO:
prover-client:proving-broker Proving Broker started
14:23:56 [14:23:56.874]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:146131359e0c58e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:146131359e0c58e3"}
14:23:56 [14:23:56.875]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8WoE3M/1 with map size 1073741824
14:23:56 [14:23:56.880]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:56 [14:23:56.906]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8baa96d9c61651fd epochNumber=2
{"provingJobId":"1:BASE_PARITY:8baa96d9c61651fd"}
14:23:56 [14:23:56.907]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8WoE3M/2 with map size 1073741824
14:23:56 [14:23:56.910]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:57 [14:23:57.930]
INFO:
prover-client:proving-broker Proving Broker started
14:23:57 [14:23:57.932]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:146131359e0c58e3 settled=false
{"provingJobId":"1:BASE_PARITY:146131359e0c58e3","status":"pending"}
14:23:57 [14:23:57.933]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8baa96d9c61651fd settled=false
{"provingJobId":"1:BASE_PARITY:8baa96d9c61651fd","status":"pending"}
14:23:57 [14:23:57.934]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:146131359e0c58e3 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:146131359e0c58e3"}
14:23:57 [14:23:57.938]
INFO:
prover-client:proving-broker Proving Broker started
14:23:57 [14:23:57.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a11bb6393764a363 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a11bb6393764a363"}
14:23:57 [14:23:57.940]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testC7q5Tr/1 with map size 1073741824
14:23:57 [14:23:57.943]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:57 [14:23:57.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48ca1aeef88203bb epochNumber=2
{"provingJobId":"1:BASE_PARITY:48ca1aeef88203bb"}
14:23:57 [14:23:57.965]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testC7q5Tr/2 with map size 1073741824
14:23:57 [14:23:57.967]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:58 [14:23:58.990]
INFO:
prover-client:proving-broker Proving Broker started
14:23:58 [14:23:58.999]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a11bb6393764a363 settled=false
{"provingJobId":"1:BASE_PARITY:a11bb6393764a363","status":"pending"}
14:23:59 [14:23:59.004]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:48ca1aeef88203bb settled=false
{"provingJobId":"1:BASE_PARITY:48ca1aeef88203bb","status":"pending"}
14:23:59 [14:23:59.007]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a11bb6393764a363 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a11bb6393764a363"}
14:23:59 [14:23:59.008]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a11bb6393764a363 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a11bb6393764a363"}
14:23:59 [14:23:59.022]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.024]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d6d3dd545148f0a epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d6d3dd545148f0a"}
14:23:59 [14:23:59.026]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0gjU0P/1 with map size 1073741824
14:23:59 [14:23:59.028]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.049]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9397a011a16425cb epochNumber=2
{"provingJobId":"1:BASE_PARITY:9397a011a16425cb"}
14:23:59 [14:23:59.050]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0gjU0P/2 with map size 1073741824
14:23:59 [14:23:59.052]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.071]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0d6d3dd545148f0a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0d6d3dd545148f0a"}
14:23:59 [14:23:59.078]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9397a011a16425cb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9397a011a16425cb"}
14:23:59 [14:23:59.092]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.093]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c17a2e48d112a47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c17a2e48d112a47"}
14:23:59 [14:23:59.094]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEUcJ6A/1 with map size 1073741824
14:23:59 [14:23:59.099]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.118]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4402257411fbd80c epochNumber=2
{"provingJobId":"1:BASE_PARITY:4402257411fbd80c"}
14:23:59 [14:23:59.120]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testEUcJ6A/2 with map size 1073741824
14:23:59 [14:23:59.124]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.141]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1c17a2e48d112a47 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1c17a2e48d112a47"}
14:23:59 [14:23:59.142]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1c17a2e48d112a47 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1c17a2e48d112a47"}
14:23:59 [14:23:59.149]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4402257411fbd80c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4402257411fbd80c"}
14:23:59 [14:23:59.163]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.165]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f1efd0745ed59ebd err=test error
{"provingJoId":"1:BASE_PARITY:f1efd0745ed59ebd"}
14:23:59 [14:23:59.173]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.175]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:61c79ed61683eea0 not found
{"provingJobId":"1:BASE_PARITY:61c79ed61683eea0"}
14:23:59 [14:23:59.179]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.180]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4bfb5825569b7d7a epochNumber=1
{"provingJobId":"1:BASE_PARITY:4bfb5825569b7d7a"}
14:23:59 [14:23:59.181]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3S7q9M/1 with map size 1073741824
14:23:59 [14:23:59.183]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.213]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.214]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e3198d7c0036caf epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e3198d7c0036caf"}
14:23:59 [14:23:59.215]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKIYVNh/1 with map size 1073741824
14:23:59 [14:23:59.217]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.340]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7e3198d7c0036caf timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:7e3198d7c0036caf"}
14:23:59 [14:23:59.347]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.349]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3901bf9207eeefb4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3901bf9207eeefb4"}
14:23:59 [14:23:59.350]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEb2haU/1 with map size 1073741824
14:23:59 [14:23:59.353]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.478]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3901bf9207eeefb4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:3901bf9207eeefb4"}
14:23:59 [14:23:59.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5162960b41ed4bf epochNumber=10
{"provingJobId":"1:BASE_PARITY:e5162960b41ed4bf"}
14:23:59 [14:23:59.485]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testEb2haU/10 with map size 1073741824
14:23:59 [14:23:59.488]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.531]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:23:59 [14:23:59.532]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:23:59 [14:23:59.650]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.651]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e79d109b638e4dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e79d109b638e4dd"}
14:23:59 [14:23:59.652]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSy6hhc/1 with map size 1073741824
14:23:59 [14:23:59.653]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.828]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9e79d109b638e4dd timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9e79d109b638e4dd"}
14:23:59 [14:23:59.881]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3284a77b2d984842 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3284a77b2d984842"}
14:23:59 [14:23:59.884]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testI7mTo2/1 with map size 1073741824
14:23:59 [14:23:59.887]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.914]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3284a77b2d984842 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3284a77b2d984842"}
14:23:59 [14:23:59.920]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.921]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bec399def2738162 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bec399def2738162"}
14:23:59 [14:23:59.923]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvBJWHe/1 with map size 1073741824
14:23:59 [14:23:59.926]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.948]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bec399def2738162 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:bec399def2738162"}
14:23:59 [14:23:59.950]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bec399def2738162 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:bec399def2738162"}
14:23:59 [14:23:59.964]
INFO:
prover-client:proving-broker Proving Broker started
14:23:59 [14:23:59.965]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c2fd9b515c12858 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c2fd9b515c12858"}
14:23:59 [14:23:59.966]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGVm7XX/1 with map size 1073741824
14:23:59 [14:23:59.970]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:23:59 [14:23:59.994]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4c2fd9b515c12858 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4c2fd9b515c12858"}
14:24:00 [14:24:00.013]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.015]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa5dab4531f83971 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa5dab4531f83971"}
14:24:00 [14:24:00.017]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3NzQyg/1 with map size 1073741824
14:24:00 [14:24:00.022]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.040]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fa5dab4531f83971 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:fa5dab4531f83971"}
14:24:00 [14:24:00.041]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b4454e3d91ba93b epochNumber=3
{"provingJobId":"1:BASE_PARITY:0b4454e3d91ba93b"}
14:24:00 [14:24:00.043]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test3NzQyg/3 with map size 1073741824
14:24:00 [14:24:00.047]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.068]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:00 [14:24:00.069]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:24:00 [14:24:00.093]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:fa5dab4531f83971 err=test error
{"provingJoId":"1:BASE_PARITY:fa5dab4531f83971"}
14:24:00 [14:24:00.102]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf3Iu2K/1 with map size 1073741824
14:24:00 [14:24:00.108]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.129]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testf3Iu2K/2 with map size 1073741824
14:24:00 [14:24:00.135]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.156]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.159]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d4fa471a3a983b63 settled=false
{"provingJobId":"1:BASE_PARITY:d4fa471a3a983b63","status":"pending"}
14:24:00 [14:24:00.161]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:75ffccfe7cac65c6 settled=false
{"provingJobId":"1:BASE_PARITY:75ffccfe7cac65c6","status":"pending"}
14:24:00 [14:24:00.169]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoJHUbA/1 with map size 1073741824
14:24:00 [14:24:00.172]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.202]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testoJHUbA/2 with map size 1073741824
14:24:00 [14:24:00.207]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.236]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.240]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:76973a476abbe3a6 settled=true
{"provingJobId":"1:BASE_PARITY:76973a476abbe3a6","status":"fulfilled"}
14:24:00 [14:24:00.243]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:20853bb3b968fa83 settled=true
{"provingJobId":"2:BASE_PARITY:20853bb3b968fa83","status":"fulfilled"}
14:24:00 [14:24:00.250]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5lCUPQ/1 with map size 1073741824
14:24:00 [14:24:00.253]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.277]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5lCUPQ/2 with map size 1073741824
14:24:00 [14:24:00.280]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.306]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.310]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:36d2c66aefaf1590 settled=true
{"provingJobId":"1:BASE_PARITY:36d2c66aefaf1590","status":"fulfilled"}
14:24:00 [14:24:00.312]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7eedb2a50b032108 settled=false
{"provingJobId":"1:BASE_PARITY:7eedb2a50b032108","status":"pending"}
14:24:00 [14:24:00.319]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.321]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2838806a14305ff6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2838806a14305ff6"}
14:24:00 [14:24:00.322]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9taGWy/1 with map size 1073741824
14:24:00 [14:24:00.325]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.348]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.348]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54df8872ecbaa9b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:54df8872ecbaa9b8"}
14:24:00 [14:24:00.350]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:54df8872ecbaa9b8: Error: db error: Error: db error
14:24:00 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:54df8872ecbaa9b8"}
14:24:00 [14:24:00.355]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.355]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a2613dc460181fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a2613dc460181fe"}
14:24:00 [14:24:00.356]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPTYzsx/1 with map size 1073741824
14:24:00 [14:24:00.358]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.375]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7a2613dc460181fe type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7a2613dc460181fe"}
14:24:00 [14:24:00.376]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7a2613dc460181fe type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7a2613dc460181fe"}
14:24:00 [14:24:00.386]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.387]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89673e95ea705e21 epochNumber=1
{"provingJobId":"1:BASE_PARITY:89673e95ea705e21"}
14:24:00 [14:24:00.388]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdnbuxJ/1 with map size 1073741824
14:24:00 [14:24:00.390]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.416]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:89673e95ea705e21 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:89673e95ea705e21"}
14:24:00 [14:24:00.417]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:89673e95ea705e21 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:89673e95ea705e21"}
14:24:00 [14:24:00.417]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:89673e95ea705e21: Error: db error
14:24:00 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:89673e95ea705e21"}
14:24:00 [14:24:00.424]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.425]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a5948cc850317605 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a5948cc850317605"}
14:24:00 [14:24:00.426]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvO97r1/1 with map size 1073741824
14:24:00 [14:24:00.428]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.449]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a5948cc850317605 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a5948cc850317605"}
14:24:00 [14:24:00.450]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a5948cc850317605 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a5948cc850317605"}
14:24:00 [14:24:00.460]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.461]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88381995ba8a4e01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:88381995ba8a4e01"}
14:24:00 [14:24:00.462]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgSYg9Q/1 with map size 1073741824
14:24:00 [14:24:00.466]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.486]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:88381995ba8a4e01 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:88381995ba8a4e01"}
14:24:00 [14:24:00.487]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:88381995ba8a4e01 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:88381995ba8a4e01"}
14:24:00 [14:24:00.488]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:88381995ba8a4e01 jobErr=test error: Error: db error
14:24:00 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:88381995ba8a4e01"}
14:24:00 [14:24:00.496]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.497]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ee23ff1d90988491 not found
{"provingJobId":"1:BASE_PARITY:ee23ff1d90988491"}
14:24:00 [14:24:00.516]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.517]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:afddbe63d9aefa69 err=test error
{"provingJoId":"1:BASE_PARITY:afddbe63d9aefa69"}
14:24:00 [14:24:00.522]
INFO:
prover-client:proving-broker Proving Broker started
14:24:00 [14:24:00.533]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0dd6f9fbe8c0cd63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0dd6f9fbe8c0cd63"}
14:24:00 [14:24:00.536]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKxEkPo/1 with map size 1073741824
14:24:00 [14:24:00.540]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.557]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0dd6f9fbe8c0cd63 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0dd6f9fbe8c0cd63"}
14:24:00 [14:24:00.558]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0dd6f9fbe8c0cd63 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0dd6f9fbe8c0cd63"}
14:24:00 [14:24:00.575]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:2e477133683d93ed epochNumber=2
{"provingJobId":"2:BASE_PARITY:2e477133683d93ed"}
14:24:00 [14:24:00.576]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKxEkPo/2 with map size 1073741824
14:24:00 [14:24:00.582]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.606]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:2e477133683d93ed type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:2e477133683d93ed"}
14:24:00 [14:24:00.607]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2e477133683d93ed type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:2e477133683d93ed"}
14:24:00 [14:24:00.674]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:b428b30229008b19 epochNumber=3
{"provingJobId":"3:BASE_PARITY:b428b30229008b19"}
14:24:00 [14:24:00.675]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testKxEkPo/3 with map size 1073741824
14:24:00 [14:24:00.680]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.701]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:00 [14:24:00.702]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:24:00 [14:24:00.759]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:cbf7571ef9fe14b6 epochNumber=4
{"provingJobId":"4:BASE_PARITY:cbf7571ef9fe14b6"}
14:24:00 [14:24:00.760]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testKxEkPo/4 with map size 1073741824
14:24:00 [14:24:00.762]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.781]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:00 [14:24:00.781]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
14:24:00 [14:24:00.841]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:1df1188cc1be6468 epochNumber=5
{"provingJobId":"5:BASE_PARITY:1df1188cc1be6468"}
14:24:00 [14:24:00.843]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testKxEkPo/5 with map size 1073741824
14:24:00 [14:24:00.845]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:24:00 [14:24:00.887]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:24:00 [14:24:00.888]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
14:24:00
PASS src/proving_broker/proving_broker.test.ts (
13.231 s)
14:24:00 ProvingBroker
14:24:00 Producer API
14:24:00
✓ refuses stale jobs (44 ms)
14:24:00
✓ enqueues jobs (4 ms)
14:24:00
✓ ignores duplicate jobs (3 ms)
14:24:00
✓ reports correct status when enqueuing repeat jobs (3 ms)
14:24:00
✓ throws an error in case of duplicate job IDs (24 ms)
14:24:00
✓ returns not-found status for non-existing jobs (1 ms)
14:24:00
✓ cancels jobs in queue (3 ms)
14:24:00
✓ cancels jobs in-progress (2 ms)
14:24:00
✓ returns job result if successful (3 ms)
14:24:00
✓ returns job error if failed (2 ms)
14:24:00
✓ correctly returns job status for concurrent writes (3 ms)
14:24:00
✓ refuses stale jobs (34 ms)
14:24:00
✓ enqueues jobs (31 ms)
14:24:00
✓ ignores duplicate jobs (28 ms)
14:24:00
✓ reports correct status when enqueuing repeat jobs (30 ms)
14:24:00
✓ throws an error in case of duplicate job IDs (22 ms)
14:24:00
✓ returns not-found status for non-existing jobs (2 ms)
14:24:00
✓ cancels jobs in queue (31 ms)
14:24:00
✓ cancels jobs in-progress (34 ms)
14:24:00
✓ returns job result if successful (29 ms)
14:24:00
✓ returns job error if failed (29 ms)
14:24:00
✓ correctly returns job status for concurrent writes (33 ms)
14:24:00 Consumer API
14:24:00
✓ returns undefined if no jobs are available (1 ms)
14:24:00
✓ returns jobs in priority order (2 ms)
14:24:00
✓ returns undefined if no jobs are available for the given allowList (2 ms)
14:24:00
✓ returns a job if it is in the allowList (2 ms)
14:24:00
✓ returns the most important job if it is in the allowList (10 ms)
14:24:00
✓ returns any job if filter is empty (3 ms)
14:24:00
✓ returns a new job when reporting job success (2 ms)
14:24:00
✓ returns a new job when reporting permanent error (3 ms)
14:24:00
✓ returns a new job when reporting retry-able error (2 ms)
14:24:00
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
14:24:00
✓ returns a new job if job is already in progress elsewhere (1060 ms)
14:24:00
✓ avoids sending the same job to a new agent after a restart (1008 ms)
14:24:00
✓ avoids sending a completed job to a new agent after a restart (1007 ms)
14:24:00
✓ tracks job result if in progress (4 ms)
14:24:00
✓ tracks job result even if job is in queue (4 ms)
14:24:00
✓ ignores reported job error if unknown job (2 ms)
14:24:00
✓ ignores job result if unknown job (1 ms)
14:24:00
✓ returns undefined if no jobs are available (2 ms)
14:24:00
✓ returns jobs in priority order (42 ms)
14:24:00
✓ returns undefined if no jobs are available for the given allowList (23 ms)
14:24:00
✓ returns a job if it is in the allowList (51 ms)
14:24:00
✓ returns the most important job if it is in the allowList (62 ms)
14:24:00
✓ returns any job if filter is empty (55 ms)
14:24:00
✓ returns a new job when reporting job success (40 ms)
14:24:00
✓ returns a new job when reporting permanent error (37 ms)
14:24:00
✓ returns a new job when reporting retry-able error (39 ms)
14:24:00
✓ returns a new job when reporting progress if current one is cancelled (39 ms)
14:24:00
✓ returns a new job if job is already in progress elsewhere (1116 ms)
14:24:00
✓ avoids sending the same job to a new agent after a restart (1066 ms)
14:24:00
✓ avoids sending a completed job to a new agent after a restart (1081 ms)
14:24:00
✓ tracks job result if in progress (69 ms)
14:24:00
✓ tracks job result even if job is in queue (70 ms)
14:24:00
✓ ignores reported job error if unknown job (8 ms)
14:24:00
✓ ignores job result if unknown job (7 ms)
14:24:00 Timeouts
14:24:00
✓ tracks in progress jobs (3 ms)
14:24:00
✓ re-enqueues jobs that time out (109 ms)
14:24:00
✓ cancel stale jobs that time out (237 ms)
14:24:00
✓ keeps the jobs in progress while it is alive (205 ms)
14:24:00
✓ tracks in progress jobs (32 ms)
14:24:00
✓ re-enqueues jobs that time out (135 ms)
14:24:00
✓ cancel stale jobs that time out (303 ms)
14:24:00
✓ keeps the jobs in progress while it is alive (231 ms)
14:24:00 Retries
14:24:00
✓ retries jobs (3 ms)
14:24:00
✓ retries up to a maximum number of times (3 ms)
14:24:00
✓ passing retry=false does not retry (2 ms)
14:24:00
✓ does not retry if job is stale (27 ms)
14:24:00
✓ retries jobs (38 ms)
14:24:00
✓ retries up to a maximum number of times (42 ms)
14:24:00
✓ passing retry=false does not retry (45 ms)
14:24:00
✓ does not retry if job is stale (88 ms)
14:24:00 Database management
14:24:00
✓ re-enqueues proof requests on start (3 ms)
14:24:00
✓ restores proof results on start (2 ms)
14:24:00
✓ only re-enqueues unfinished jobs (2 ms)
14:24:00
✓ saves job when enqueued (3 ms)
14:24:00
✓ does not retain job if database fails to save (2 ms)
14:24:00
✓ saves job result (2 ms)
14:24:00
✓ saves result even if database fails to save (2 ms)
14:24:00
✓ saves job error (2 ms)
14:24:00
✓ saves job error even if database fails to save (2 ms)
14:24:00
✓ does not save job result if job is unknown (1 ms)
14:24:00
✓ does not save job error if job is unknown (5 ms)
14:24:00
✓ cleans up old jobs periodically (263 ms)
14:24:00
✓ re-enqueues proof requests on start (67 ms)
14:24:00
✓ restores proof results on start (83 ms)
14:24:00
✓ only re-enqueues unfinished jobs (68 ms)
14:24:00
✓ saves job when enqueued (27 ms)
14:24:00
✓ does not retain job if database fails to save (6 ms)
14:24:00
✓ saves job result (31 ms)
14:24:00
✓ saves result even if database fails to save (38 ms)
14:24:00
✓ saves job error (35 ms)
14:24:00
✓ saves job error even if database fails to save (33 ms)
14:24:00
✓ does not save job result if job is unknown (9 ms)
14:24:00
✓ does not save job error if job is unknown (16 ms)
14:24:00
✓ cleans up old jobs periodically (401 ms)
14:24:00
14:24:00
Test Suites: 1 passed, 1 total
14:24:00
Tests: 96 passed, 96 total
14:24:00
Snapshots: 0 total
14:24:00
Time: 13.327 s
14:24:00
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
14:24:00
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?