Parent Log:
http://ci.aztec-labs.com/30965617224928f2
Command: 1574a02ff8f30339 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/1677dac936d084443e64d6c4ba6e45d3958ba1f8
Env: REF_NAME=gh-readonly-queue/next/pr-15122-df0c3e8efd6211cf15eedf67d2a591cc54099ac8 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Wed Jun 18 18:31:33 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15122_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
18:31:37 [18:31:37.727]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.736]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15677e790345a4ec epochNumber=42
{"provingJobId":"1:BASE_PARITY:15677e790345a4ec"}
18:31:37 [18:31:37.740]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:2ec1eee9333f5e17 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ec1eee9333f5e17"}
18:31:37 [18:31:37.775]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a44e57657f7d143a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a44e57657f7d143a"}
18:31:37 [18:31:37.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a64d99b6fba1a42a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a64d99b6fba1a42a"}
18:31:37 [18:31:37.779]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.780]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b1be215c440537c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b1be215c440537c2"}
18:31:37 [18:31:37.781]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b1be215c440537c2 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b1be215c440537c2"}
18:31:37 [18:31:37.783]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.784]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0292012c48333128 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0292012c48333128"}
18:31:37 [18:31:37.785]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0292012c48333128 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0292012c48333128"}
18:31:37 [18:31:37.786]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0292012c48333128 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0292012c48333128"}
18:31:37 [18:31:37.786]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0292012c48333128 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0292012c48333128"}
18:31:37 [18:31:37.787]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.788]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c0f980512bd40d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c0f980512bd40d9"}
18:31:37 [18:31:37.816]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.818]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.819]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5aaa3e82e8757dcd epochNumber=1
{"provingJobId":"1:BASE_PARITY:5aaa3e82e8757dcd"}
18:31:37 [18:31:37.821]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5aaa3e82e8757dcd
{"provingJobId":"1:BASE_PARITY:5aaa3e82e8757dcd"}
18:31:37 [18:31:37.821]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5aaa3e82e8757dcd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5aaa3e82e8757dcd"}
18:31:37 [18:31:37.822]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5aaa3e82e8757dcd type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5aaa3e82e8757dcd"}
18:31:37 [18:31:37.824]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.825]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48cc5ea83304db6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:48cc5ea83304db6c"}
18:31:37 [18:31:37.827]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:48cc5ea83304db6c
{"provingJobId":"1:BASE_PARITY:48cc5ea83304db6c"}
18:31:37 [18:31:37.827]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:48cc5ea83304db6c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:48cc5ea83304db6c"}
18:31:37 [18:31:37.830]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.831]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c7a3d13becc3301 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c7a3d13becc3301"}
18:31:37 [18:31:37.832]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3c7a3d13becc3301 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3c7a3d13becc3301"}
18:31:37 [18:31:37.832]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3c7a3d13becc3301 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3c7a3d13becc3301"}
18:31:37 [18:31:37.835]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.837]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3893a859f4fa318f epochNumber=1
{"provingJobId":"1:BASE_PARITY:3893a859f4fa318f"}
18:31:37 [18:31:37.837]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3893a859f4fa318f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3893a859f4fa318f"}
18:31:37 [18:31:37.838]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3893a859f4fa318f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3893a859f4fa318f"}
18:31:37 [18:31:37.840]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.840]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a56ecd1faf4af266 epochNumber=0
{"provingJobId":"1:BASE_PARITY:a56ecd1faf4af266"}
18:31:37 [18:31:37.841]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a56ecd1faf4af266 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:a56ecd1faf4af266"}
18:31:37 [18:31:37.841]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8969f33a553053a3 epochNumber=0
{"provingJobId":"1:BASE_PARITY:8969f33a553053a3"}
18:31:37 [18:31:37.841]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a56ecd1faf4af266 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:a56ecd1faf4af266"}
18:31:37 [18:31:37.842]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e42cb15706b6aca9 epochNumber=0
{"provingJobId":"1:BASE_PARITY:e42cb15706b6aca9"}
18:31:37 [18:31:37.843]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.855]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.856]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08f5451f3efb9616 epochNumber=2
{"provingJobId":"1:BASE_PARITY:08f5451f3efb9616"}
18:31:37 [18:31:37.857]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a51d3fe9d622370 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a51d3fe9d622370"}
18:31:37 [18:31:37.858]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.859]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:accde9041074cbb0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:accde9041074cbb0"}
18:31:37 [18:31:37.860]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.860]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cb4a755ab38a40d epochNumber=1
{"provingJobId":"1:BASE_PARITY:0cb4a755ab38a40d"}
18:31:37 [18:31:37.860]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7c8c3056ae8370a epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7c8c3056ae8370a"}
18:31:37 [18:31:37.861]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d42897f98193ed8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5d42897f98193ed8"}
18:31:37 [18:31:37.861]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61a0aa30c6d1dffb epochNumber=1
{"provingJobId":"1:BASE_PARITY:61a0aa30c6d1dffb"}
18:31:37 [18:31:37.862]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.863]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a89df93da8e3454 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a89df93da8e3454"}
18:31:37 [18:31:37.863]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df006cfba49c05b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df006cfba49c05b1"}
18:31:37 [18:31:37.863]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c56b949d5146155 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9c56b949d5146155"}
18:31:37 [18:31:37.863]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c702223d1257ee0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c702223d1257ee0"}
18:31:37 [18:31:37.864]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.865]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00b0a281520b5f47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:00b0a281520b5f47"}
18:31:37 [18:31:37.865]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff5263235972bec0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff5263235972bec0"}
18:31:37 [18:31:37.865]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ad162bac0b499e6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5ad162bac0b499e6"}
18:31:37 [18:31:37.865]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a44c1f23002bac41 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a44c1f23002bac41"}
18:31:37 [18:31:37.866]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.867]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61b5609aa053adf7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:61b5609aa053adf7"}
18:31:37 [18:31:37.867]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ab6aace0c4103a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ab6aace0c4103a0"}
18:31:37 [18:31:37.867]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:61b5609aa053adf7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:61b5609aa053adf7"}
18:31:37 [18:31:37.868]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.869]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c3d1e66e63e7f1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c3d1e66e63e7f1f"}
18:31:37 [18:31:37.869]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a19e268bb081c7c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a19e268bb081c7c6"}
18:31:37 [18:31:37.869]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5c3d1e66e63e7f1f type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:5c3d1e66e63e7f1f"}
18:31:37 [18:31:37.870]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.871]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:335ea838ded68f6e epochNumber=1
{"provingJobId":"1:BASE_PARITY:335ea838ded68f6e"}
18:31:37 [18:31:37.871]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e686b29edcc57584 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e686b29edcc57584"}
18:31:37 [18:31:37.871]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:335ea838ded68f6e type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:335ea838ded68f6e"}
18:31:37 [18:31:37.873]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f6d94332b0008c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0f6d94332b0008c2"}
18:31:37 [18:31:37.874]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0f6d94332b0008c2
{"provingJobId":"1:BASE_PARITY:0f6d94332b0008c2"}
18:31:37 [18:31:37.874]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0f6d94332b0008c2 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:0f6d94332b0008c2"}
18:31:37 [18:31:37.874]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ad04decb888acca epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ad04decb888acca"}
18:31:37 [18:31:37.874]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0f6d94332b0008c2 has already been completed
{"provingJobId":"1:BASE_PARITY:0f6d94332b0008c2"}
18:31:37 [18:31:37.875]
INFO:
prover-client:proving-broker Proving Broker started
18:31:37 [18:31:37.876]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:127d366ccc9de9c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:127d366ccc9de9c6"}
18:31:37 [18:31:37.876]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25b03779d2e1a97d epochNumber=2
{"provingJobId":"1:BASE_PARITY:25b03779d2e1a97d"}
18:31:38 [18:31:38.929]
INFO:
prover-client:proving-broker Proving Broker started
18:31:38 [18:31:38.930]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:127d366ccc9de9c6 settled=false
{"provingJobId":"1:BASE_PARITY:127d366ccc9de9c6","status":"pending"}
18:31:38 [18:31:38.930]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:25b03779d2e1a97d settled=false
{"provingJobId":"1:BASE_PARITY:25b03779d2e1a97d","status":"pending"}
18:31:38 [18:31:38.931]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:127d366ccc9de9c6 type=BASE_PARITY startedAt=1750271497876 older agent has taken job
{"provingJobId":"1:BASE_PARITY:127d366ccc9de9c6"}
18:31:38 [18:31:38.932]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:127d366ccc9de9c6 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:127d366ccc9de9c6"}
18:31:38 [18:31:38.933]
INFO:
prover-client:proving-broker Proving Broker started
18:31:38 [18:31:38.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68597428d73d2fb1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:68597428d73d2fb1"}
18:31:38 [18:31:38.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c951734c985ea685 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c951734c985ea685"}
18:31:39 [18:31:39.936]
INFO:
prover-client:proving-broker Proving Broker started
18:31:39 [18:31:39.937]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:68597428d73d2fb1 settled=false
{"provingJobId":"1:BASE_PARITY:68597428d73d2fb1","status":"pending"}
18:31:39 [18:31:39.937]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c951734c985ea685 settled=false
{"provingJobId":"1:BASE_PARITY:c951734c985ea685","status":"pending"}
18:31:39 [18:31:39.938]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:68597428d73d2fb1 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:68597428d73d2fb1"}
18:31:39 [18:31:39.939]
INFO:
prover-client:proving-broker Proving Broker started
18:31:39 [18:31:39.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34a8bf46e2cc2636 epochNumber=1
{"provingJobId":"1:BASE_PARITY:34a8bf46e2cc2636"}
18:31:39 [18:31:39.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:49d61244553328a0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:49d61244553328a0"}
18:31:40 [18:31:40.941]
INFO:
prover-client:proving-broker Proving Broker started
18:31:40 [18:31:40.941]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:34a8bf46e2cc2636 settled=false
{"provingJobId":"1:BASE_PARITY:34a8bf46e2cc2636","status":"pending"}
18:31:40 [18:31:40.941]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:49d61244553328a0 settled=false
{"provingJobId":"1:BASE_PARITY:49d61244553328a0","status":"pending"}
18:31:40 [18:31:40.942]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:34a8bf46e2cc2636 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:34a8bf46e2cc2636"}
18:31:40 [18:31:40.942]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:34a8bf46e2cc2636 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:34a8bf46e2cc2636"}
18:31:40 [18:31:40.943]
INFO:
prover-client:proving-broker Proving Broker started
18:31:40 [18:31:40.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2330f29f49959eec epochNumber=1
{"provingJobId":"1:BASE_PARITY:2330f29f49959eec"}
18:31:40 [18:31:40.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a1b6a6ad00839fc epochNumber=2
{"provingJobId":"1:BASE_PARITY:1a1b6a6ad00839fc"}
18:31:40 [18:31:40.945]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2330f29f49959eec type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2330f29f49959eec"}
18:31:40 [18:31:40.945]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1a1b6a6ad00839fc type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1a1b6a6ad00839fc"}
18:31:40 [18:31:40.946]
INFO:
prover-client:proving-broker Proving Broker started
18:31:40 [18:31:40.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b82e9fd795c5b4c epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b82e9fd795c5b4c"}
18:31:40 [18:31:40.947]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64bf97a8a560b9ab epochNumber=2
{"provingJobId":"1:BASE_PARITY:64bf97a8a560b9ab"}
18:31:40 [18:31:40.947]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8b82e9fd795c5b4c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8b82e9fd795c5b4c"}
18:31:40 [18:31:40.947]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8b82e9fd795c5b4c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8b82e9fd795c5b4c"}
18:31:40 [18:31:40.947]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:64bf97a8a560b9ab type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:64bf97a8a560b9ab"}
18:31:40 [18:31:40.948]
INFO:
prover-client:proving-broker Proving Broker started
18:31:40 [18:31:40.949]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0cc8231c487a5819 err=test error
{"provingJoId":"1:BASE_PARITY:0cc8231c487a5819"}
18:31:40 [18:31:40.949]
INFO:
prover-client:proving-broker Proving Broker started
18:31:40 [18:31:40.950]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2f71abd2c4142aa0 not found
{"provingJobId":"1:BASE_PARITY:2f71abd2c4142aa0"}
18:31:40 [18:31:40.951]
INFO:
prover-client:proving-broker Proving Broker started
18:31:40 [18:31:40.951]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c3f2ad1dd2b3cd1d epochNumber=1
{"provingJobId":"1:BASE_PARITY:c3f2ad1dd2b3cd1d"}
18:31:40 [18:31:40.953]
INFO:
prover-client:proving-broker Proving Broker started
18:31:40 [18:31:40.953]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a6edb33b0c2227d epochNumber=1
{"provingJobId":"1:BASE_PARITY:6a6edb33b0c2227d"}
18:31:41 [18:31:41.054]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6a6edb33b0c2227d timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6a6edb33b0c2227d"}
18:31:41 [18:31:41.061]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.062]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a76d420803729ab epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a76d420803729ab"}
18:31:41 [18:31:41.163]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2a76d420803729ab timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2a76d420803729ab"}
18:31:41 [18:31:41.165]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54cc45b59b6c2bc2 epochNumber=10
{"provingJobId":"1:BASE_PARITY:54cc45b59b6c2bc2"}
18:31:41 [18:31:41.189]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:41 [18:31:41.292]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.293]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42b090af323f0ad8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:42b090af323f0ad8"}
18:31:41 [18:31:41.447]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:42b090af323f0ad8 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:42b090af323f0ad8"}
18:31:41 [18:31:41.501]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.503]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f05b7395878d4581 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f05b7395878d4581"}
18:31:41 [18:31:41.504]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f05b7395878d4581 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f05b7395878d4581"}
18:31:41 [18:31:41.506]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.507]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea771ee3be35fcc3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea771ee3be35fcc3"}
18:31:41 [18:31:41.508]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ea771ee3be35fcc3 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:ea771ee3be35fcc3"}
18:31:41 [18:31:41.509]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ea771ee3be35fcc3 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:ea771ee3be35fcc3"}
18:31:41 [18:31:41.510]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.511]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e6aada5ae144d66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e6aada5ae144d66"}
18:31:41 [18:31:41.512]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9e6aada5ae144d66 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9e6aada5ae144d66"}
18:31:41 [18:31:41.513]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2abd11be5842e2b epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2abd11be5842e2b"}
18:31:41 [18:31:41.514]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f2abd11be5842e2b type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f2abd11be5842e2b"}
18:31:41 [18:31:41.515]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a7a6bab6bc128b16 epochNumber=3
{"provingJobId":"1:BASE_PARITY:a7a6bab6bc128b16"}
18:31:41 [18:31:41.538]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:41 [18:31:41.540]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f2abd11be5842e2b err=test error
{"provingJoId":"1:BASE_PARITY:f2abd11be5842e2b"}
18:31:41 [18:31:41.543]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.543]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f1356f8ff5fcd068 settled=false
{"provingJobId":"1:BASE_PARITY:f1356f8ff5fcd068","status":"pending"}
18:31:41 [18:31:41.543]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:221dd907a3e3602b settled=false
{"provingJobId":"1:BASE_PARITY:221dd907a3e3602b","status":"pending"}
18:31:41 [18:31:41.545]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.545]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d50af737dd7656e3 settled=true
{"provingJobId":"1:BASE_PARITY:d50af737dd7656e3","status":"fulfilled"}
18:31:41 [18:31:41.545]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:fab66b8f7e2daec8 settled=true
{"provingJobId":"2:BASE_PARITY:fab66b8f7e2daec8","status":"fulfilled"}
18:31:41 [18:31:41.546]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.546]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fe31820c52781f96 settled=true
{"provingJobId":"1:BASE_PARITY:fe31820c52781f96","status":"fulfilled"}
18:31:41 [18:31:41.547]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:97ea9311bafbf97c settled=false
{"provingJobId":"1:BASE_PARITY:97ea9311bafbf97c","status":"pending"}
18:31:41 [18:31:41.548]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.549]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40493425783a48af epochNumber=1
{"provingJobId":"1:BASE_PARITY:40493425783a48af"}
18:31:41 [18:31:41.550]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.550]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8867d374827d0a0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:8867d374827d0a0b"}
18:31:41 [18:31:41.551]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:8867d374827d0a0b: Error: db error: Error: db error
18:31:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:8867d374827d0a0b"}
18:31:41 [18:31:41.553]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.553]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e6f3de22e476ce9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e6f3de22e476ce9"}
18:31:41 [18:31:41.553]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0e6f3de22e476ce9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0e6f3de22e476ce9"}
18:31:41 [18:31:41.554]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0e6f3de22e476ce9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0e6f3de22e476ce9"}
18:31:41 [18:31:41.555]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.556]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0581459f5b0094da epochNumber=1
{"provingJobId":"1:BASE_PARITY:0581459f5b0094da"}
18:31:41 [18:31:41.556]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0581459f5b0094da type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0581459f5b0094da"}
18:31:41 [18:31:41.556]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0581459f5b0094da type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0581459f5b0094da"}
18:31:41 [18:31:41.557]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:0581459f5b0094da: Error: db error
18:31:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:0581459f5b0094da"}
18:31:41 [18:31:41.558]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.558]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a26bd0b5edba7c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9a26bd0b5edba7c4"}
18:31:41 [18:31:41.558]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9a26bd0b5edba7c4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9a26bd0b5edba7c4"}
18:31:41 [18:31:41.559]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9a26bd0b5edba7c4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9a26bd0b5edba7c4"}
18:31:41 [18:31:41.560]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.560]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ad8ba560043b4f6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ad8ba560043b4f6"}
18:31:41 [18:31:41.560]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6ad8ba560043b4f6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6ad8ba560043b4f6"}
18:31:41 [18:31:41.560]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6ad8ba560043b4f6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6ad8ba560043b4f6"}
18:31:41 [18:31:41.561]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:6ad8ba560043b4f6 jobErr=test error: Error: db error
18:31:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:6ad8ba560043b4f6"}
18:31:41 [18:31:41.562]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.562]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:619bed0f762210d0 not found
{"provingJobId":"1:BASE_PARITY:619bed0f762210d0"}
18:31:41 [18:31:41.563]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.564]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8e0229ca16180935 err=test error
{"provingJoId":"1:BASE_PARITY:8e0229ca16180935"}
18:31:41 [18:31:41.564]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.574]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d403beb5de7df323 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d403beb5de7df323"}
18:31:41 [18:31:41.575]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d403beb5de7df323 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d403beb5de7df323"}
18:31:41 [18:31:41.575]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d403beb5de7df323 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d403beb5de7df323"}
18:31:41 [18:31:41.585]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:1e4836dd34f95000 epochNumber=2
{"provingJobId":"2:BASE_PARITY:1e4836dd34f95000"}
18:31:41 [18:31:41.586]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:1e4836dd34f95000 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:1e4836dd34f95000"}
18:31:41 [18:31:41.586]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:1e4836dd34f95000 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:1e4836dd34f95000"}
18:31:41 [18:31:41.648]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:dd23bffe71d5e926 epochNumber=3
{"provingJobId":"3:BASE_PARITY:dd23bffe71d5e926"}
18:31:41 [18:31:41.671]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:41 [18:31:41.710]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:9ae46e730414ba46 epochNumber=4
{"provingJobId":"4:BASE_PARITY:9ae46e730414ba46"}
18:31:41 [18:31:41.723]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:41 [18:31:41.773]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:4528359479265069 epochNumber=5
{"provingJobId":"5:BASE_PARITY:4528359479265069"}
18:31:41 [18:31:41.776]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:41 [18:31:41.828]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.828]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36d9fb3f6f816e0f epochNumber=42
{"provingJobId":"1:BASE_PARITY:36d9fb3f6f816e0f"}
18:31:41 [18:31:41.830]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testhPnrhb/42 with map size 1073741824
18:31:41 [18:31:41.835]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:41 [18:31:41.862]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:7648a4f003274ed8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7648a4f003274ed8"}
18:31:41 [18:31:41.866]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.867]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92f5a7fea2898de7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:92f5a7fea2898de7"}
18:31:41 [18:31:41.868]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRKwACr/1 with map size 1073741824
18:31:41 [18:31:41.869]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:41 [18:31:41.887]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6943cc63b452e734 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6943cc63b452e734"}
18:31:41 [18:31:41.895]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.895]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c0f19439933303f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c0f19439933303f"}
18:31:41 [18:31:41.896]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testu2yivM/1 with map size 1073741824
18:31:41 [18:31:41.897]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:41 [18:31:41.914]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1c0f19439933303f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1c0f19439933303f"}
18:31:41 [18:31:41.928]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.929]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:483effa96260c223 epochNumber=1
{"provingJobId":"1:BASE_PARITY:483effa96260c223"}
18:31:41 [18:31:41.930]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTvpozI/1 with map size 1073741824
18:31:41 [18:31:41.932]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:41 [18:31:41.958]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:483effa96260c223 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:483effa96260c223"}
18:31:41 [18:31:41.958]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:483effa96260c223 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:483effa96260c223"}
18:31:41 [18:31:41.966]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:483effa96260c223 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:483effa96260c223"}
18:31:41 [18:31:41.970]
INFO:
prover-client:proving-broker Proving Broker started
18:31:41 [18:31:41.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:23f7b670651d9891 epochNumber=1
{"provingJobId":"1:BASE_PARITY:23f7b670651d9891"}
18:31:41 [18:31:41.972]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPr6mFa/1 with map size 1073741824
18:31:41 [18:31:41.974]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:41 [18:31:41.999]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.002]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.002]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d182a6885d9877c epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d182a6885d9877c"}
18:31:42 [18:31:42.003]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5UbnTl/1 with map size 1073741824
18:31:42 [18:31:42.005]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.024]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3d182a6885d9877c
{"provingJobId":"1:BASE_PARITY:3d182a6885d9877c"}
18:31:42 [18:31:42.025]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3d182a6885d9877c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3d182a6885d9877c"}
18:31:42 [18:31:42.025]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3d182a6885d9877c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3d182a6885d9877c"}
18:31:42 [18:31:42.034]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.035]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dee5f188016649a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dee5f188016649a8"}
18:31:42 [18:31:42.036]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhgVDyR/1 with map size 1073741824
18:31:42 [18:31:42.038]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.058]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:dee5f188016649a8
{"provingJobId":"1:BASE_PARITY:dee5f188016649a8"}
18:31:42 [18:31:42.059]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dee5f188016649a8 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:dee5f188016649a8"}
18:31:42 [18:31:42.069]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.070]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:815d1bbafb63e56a epochNumber=1
{"provingJobId":"1:BASE_PARITY:815d1bbafb63e56a"}
18:31:42 [18:31:42.071]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGq1Tuo/1 with map size 1073741824
18:31:42 [18:31:42.077]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.096]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:815d1bbafb63e56a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:815d1bbafb63e56a"}
18:31:42 [18:31:42.097]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:815d1bbafb63e56a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:815d1bbafb63e56a"}
18:31:42 [18:31:42.105]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.105]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca3eb9033e30fc00 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca3eb9033e30fc00"}
18:31:42 [18:31:42.106]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcgpVIk/1 with map size 1073741824
18:31:42 [18:31:42.108]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.129]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ca3eb9033e30fc00 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ca3eb9033e30fc00"}
18:31:42 [18:31:42.129]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ca3eb9033e30fc00 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ca3eb9033e30fc00"}
18:31:42 [18:31:42.139]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.140]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0961e6a8790ac1a9 epochNumber=0
{"provingJobId":"1:BASE_PARITY:0961e6a8790ac1a9"}
18:31:42 [18:31:42.141]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testmrFWIv/0 with map size 1073741824
18:31:42 [18:31:42.144]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.160]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0961e6a8790ac1a9 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0961e6a8790ac1a9"}
18:31:42 [18:31:42.161]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e8aa8d87ab14cd7 epochNumber=0
{"provingJobId":"1:BASE_PARITY:4e8aa8d87ab14cd7"}
18:31:42 [18:31:42.162]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0961e6a8790ac1a9 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0961e6a8790ac1a9"}
18:31:42 [18:31:42.162]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8316a7b7416df8e epochNumber=0
{"provingJobId":"1:BASE_PARITY:b8316a7b7416df8e"}
18:31:42 [18:31:42.176]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.179]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.180]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e845e6a6139ba9b4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e845e6a6139ba9b4"}
18:31:42 [18:31:42.180]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOy7Hjc/2 with map size 1073741824
18:31:42 [18:31:42.182]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.202]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72a1971238978495 epochNumber=1
{"provingJobId":"1:BASE_PARITY:72a1971238978495"}
18:31:42 [18:31:42.203]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOy7Hjc/1 with map size 1073741824
18:31:42 [18:31:42.205]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.229]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.229]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3626e391bac1cdb0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3626e391bac1cdb0"}
18:31:42 [18:31:42.230]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrXRdIn/1 with map size 1073741824
18:31:42 [18:31:42.232]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.252]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b0d81d78263a3b2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b0d81d78263a3b2"}
18:31:42 [18:31:42.254]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZ6Y3ga/1 with map size 1073741824
18:31:42 [18:31:42.256]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.273]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:500942ddb6b03f7c epochNumber=1
{"provingJobId":"1:BASE_PARITY:500942ddb6b03f7c"}
18:31:42 [18:31:42.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c51ba056b66a2748 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c51ba056b66a2748"}
18:31:42 [18:31:42.280]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZ6Y3ga/2 with map size 1073741824
18:31:42 [18:31:42.281]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.302]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae87e632dde98677 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae87e632dde98677"}
18:31:42 [18:31:42.312]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.313]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:185c503512c78bec epochNumber=1
{"provingJobId":"1:BASE_PARITY:185c503512c78bec"}
18:31:42 [18:31:42.315]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCxTJ0e/1 with map size 1073741824
18:31:42 [18:31:42.317]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3c83a2aa7fadcff epochNumber=1
{"provingJobId":"1:BASE_PARITY:e3c83a2aa7fadcff"}
18:31:42 [18:31:42.344]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:282cb2b2c779f9ef epochNumber=2
{"provingJobId":"1:BASE_PARITY:282cb2b2c779f9ef"}
18:31:42 [18:31:42.345]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCxTJ0e/2 with map size 1073741824
18:31:42 [18:31:42.348]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.370]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7359b35f4b79d8f7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7359b35f4b79d8f7"}
18:31:42 [18:31:42.381]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.381]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fdc0344d5d83cf6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0fdc0344d5d83cf6"}
18:31:42 [18:31:42.382]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXjp406/1 with map size 1073741824
18:31:42 [18:31:42.386]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.403]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:643394a7c558cf4c epochNumber=1
{"provingJobId":"1:BASE_PARITY:643394a7c558cf4c"}
18:31:42 [18:31:42.409]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:871415ac788299d0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:871415ac788299d0"}
18:31:42 [18:31:42.410]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testXjp406/2 with map size 1073741824
18:31:42 [18:31:42.414]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.435]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c80e47eb98462e1c epochNumber=1
{"provingJobId":"1:BASE_PARITY:c80e47eb98462e1c"}
18:31:42 [18:31:42.447]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf0712962f1203a2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf0712962f1203a2"}
18:31:42 [18:31:42.448]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0aDcwA/1 with map size 1073741824
18:31:42 [18:31:42.452]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.471]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:614628a0a25fec84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:614628a0a25fec84"}
18:31:42 [18:31:42.477]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bf0712962f1203a2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bf0712962f1203a2"}
18:31:42 [18:31:42.490]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.491]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bdcd9b7026855412 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bdcd9b7026855412"}
18:31:42 [18:31:42.492]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFnnMAW/1 with map size 1073741824
18:31:42 [18:31:42.494]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:380c728c640e950a epochNumber=1
{"provingJobId":"1:BASE_PARITY:380c728c640e950a"}
18:31:42 [18:31:42.519]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bdcd9b7026855412 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:bdcd9b7026855412"}
18:31:42 [18:31:42.532]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.533]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ad1eed4c0194f2d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ad1eed4c0194f2d0"}
18:31:42 [18:31:42.535]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRFgU4s/1 with map size 1073741824
18:31:42 [18:31:42.538]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.557]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a022f659e91e0e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a022f659e91e0e2"}
18:31:42 [18:31:42.563]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ad1eed4c0194f2d0 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:ad1eed4c0194f2d0"}
18:31:42 [18:31:42.567]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.568]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67e5884df70b2b5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:67e5884df70b2b5e"}
18:31:42 [18:31:42.569]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6Do8jf/1 with map size 1073741824
18:31:42 [18:31:42.571]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.590]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:67e5884df70b2b5e
{"provingJobId":"1:BASE_PARITY:67e5884df70b2b5e"}
18:31:42 [18:31:42.590]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:67e5884df70b2b5e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:67e5884df70b2b5e"}
18:31:42 [18:31:42.600]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71a20596f7f68873 epochNumber=1
{"provingJobId":"1:BASE_PARITY:71a20596f7f68873"}
18:31:42 [18:31:42.606]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:67e5884df70b2b5e has already been completed
{"provingJobId":"1:BASE_PARITY:67e5884df70b2b5e"}
18:31:42 [18:31:42.611]
INFO:
prover-client:proving-broker Proving Broker started
18:31:42 [18:31:42.612]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5147da18978bd0d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5147da18978bd0d9"}
18:31:42 [18:31:42.612]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5EwCh6/1 with map size 1073741824
18:31:42 [18:31:42.614]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:42 [18:31:42.632]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ef0f5b385808696 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2ef0f5b385808696"}
18:31:42 [18:31:42.633]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5EwCh6/2 with map size 1073741824
18:31:42 [18:31:42.644]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:43 [18:31:43.720]
INFO:
prover-client:proving-broker Proving Broker started
18:31:43 [18:31:43.727]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5147da18978bd0d9 settled=false
{"provingJobId":"1:BASE_PARITY:5147da18978bd0d9","status":"pending"}
18:31:43 [18:31:43.731]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2ef0f5b385808696 settled=false
{"provingJobId":"1:BASE_PARITY:2ef0f5b385808696","status":"pending"}
18:31:43 [18:31:43.732]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5147da18978bd0d9 type=BASE_PARITY startedAt=1750271502665 older agent has taken job
{"provingJobId":"1:BASE_PARITY:5147da18978bd0d9"}
18:31:43 [18:31:43.733]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5147da18978bd0d9 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:5147da18978bd0d9"}
18:31:43 [18:31:43.740]
INFO:
prover-client:proving-broker Proving Broker started
18:31:43 [18:31:43.742]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cda55181348f7bb epochNumber=1
{"provingJobId":"1:BASE_PARITY:0cda55181348f7bb"}
18:31:43 [18:31:43.743]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7VOb0O/1 with map size 1073741824
18:31:43 [18:31:43.746]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:43 [18:31:43.765]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d6796fce53a382e epochNumber=2
{"provingJobId":"1:BASE_PARITY:4d6796fce53a382e"}
18:31:43 [18:31:43.766]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7VOb0O/2 with map size 1073741824
18:31:43 [18:31:43.770]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:44 [18:31:44.788]
INFO:
prover-client:proving-broker Proving Broker started
18:31:44 [18:31:44.791]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0cda55181348f7bb settled=false
{"provingJobId":"1:BASE_PARITY:0cda55181348f7bb","status":"pending"}
18:31:44 [18:31:44.793]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4d6796fce53a382e settled=false
{"provingJobId":"1:BASE_PARITY:4d6796fce53a382e","status":"pending"}
18:31:44 [18:31:44.794]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0cda55181348f7bb type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:0cda55181348f7bb"}
18:31:44 [18:31:44.799]
INFO:
prover-client:proving-broker Proving Broker started
18:31:44 [18:31:44.800]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f1dbe5c89f033c5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:f1dbe5c89f033c5e"}
18:31:44 [18:31:44.801]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVYwQxq/1 with map size 1073741824
18:31:44 [18:31:44.803]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:44 [18:31:44.824]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aea1340c743766af epochNumber=2
{"provingJobId":"1:BASE_PARITY:aea1340c743766af"}
18:31:44 [18:31:44.827]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVYwQxq/2 with map size 1073741824
18:31:44 [18:31:44.832]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:45 [18:31:45.854]
INFO:
prover-client:proving-broker Proving Broker started
18:31:45 [18:31:45.869]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f1dbe5c89f033c5e settled=false
{"provingJobId":"1:BASE_PARITY:f1dbe5c89f033c5e","status":"pending"}
18:31:45 [18:31:45.872]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:aea1340c743766af settled=false
{"provingJobId":"1:BASE_PARITY:aea1340c743766af","status":"pending"}
18:31:45 [18:31:45.874]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f1dbe5c89f033c5e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f1dbe5c89f033c5e"}
18:31:45 [18:31:45.875]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f1dbe5c89f033c5e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f1dbe5c89f033c5e"}
18:31:45 [18:31:45.888]
INFO:
prover-client:proving-broker Proving Broker started
18:31:45 [18:31:45.889]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4bf5f102280b847b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4bf5f102280b847b"}
18:31:45 [18:31:45.890]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYVMXSm/1 with map size 1073741824
18:31:45 [18:31:45.892]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:45 [18:31:45.912]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:210b04bca0a6055f epochNumber=2
{"provingJobId":"1:BASE_PARITY:210b04bca0a6055f"}
18:31:45 [18:31:45.913]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYVMXSm/2 with map size 1073741824
18:31:45 [18:31:45.917]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:45 [18:31:45.940]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4bf5f102280b847b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4bf5f102280b847b"}
18:31:45 [18:31:45.948]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:210b04bca0a6055f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:210b04bca0a6055f"}
18:31:45 [18:31:45.964]
INFO:
prover-client:proving-broker Proving Broker started
18:31:45 [18:31:45.965]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22e0c0ef6b5ef6a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:22e0c0ef6b5ef6a4"}
18:31:45 [18:31:45.966]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFr25GB/1 with map size 1073741824
18:31:45 [18:31:45.973]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:45 [18:31:45.991]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ad32d09585eea79 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1ad32d09585eea79"}
18:31:45 [18:31:45.992]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFr25GB/2 with map size 1073741824
18:31:46 [18:31:46.001]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.035]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:22e0c0ef6b5ef6a4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:22e0c0ef6b5ef6a4"}
18:31:46 [18:31:46.035]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:22e0c0ef6b5ef6a4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:22e0c0ef6b5ef6a4"}
18:31:46 [18:31:46.042]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1ad32d09585eea79 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1ad32d09585eea79"}
18:31:46 [18:31:46.055]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.056]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:dd57cb72378ba313 err=test error
{"provingJoId":"1:BASE_PARITY:dd57cb72378ba313"}
18:31:46 [18:31:46.060]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.062]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1663312f90d323be not found
{"provingJobId":"1:BASE_PARITY:1663312f90d323be"}
18:31:46 [18:31:46.065]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.066]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05692cc494873fda epochNumber=1
{"provingJobId":"1:BASE_PARITY:05692cc494873fda"}
18:31:46 [18:31:46.067]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyJUrUq/1 with map size 1073741824
18:31:46 [18:31:46.071]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.094]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.095]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b3af3d0fac96c35a epochNumber=1
{"provingJobId":"1:BASE_PARITY:b3af3d0fac96c35a"}
18:31:46 [18:31:46.096]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1bGrnh/1 with map size 1073741824
18:31:46 [18:31:46.099]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.221]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b3af3d0fac96c35a timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b3af3d0fac96c35a"}
18:31:46 [18:31:46.234]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.239]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:570ea1b1263e00c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:570ea1b1263e00c9"}
18:31:46 [18:31:46.241]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvgyCEA/1 with map size 1073741824
18:31:46 [18:31:46.248]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.388]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:570ea1b1263e00c9 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:570ea1b1263e00c9"}
18:31:46 [18:31:46.390]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec5c1b3a8ac8b39b epochNumber=10
{"provingJobId":"1:BASE_PARITY:ec5c1b3a8ac8b39b"}
18:31:46 [18:31:46.391]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testvgyCEA/10 with map size 1073741824
18:31:46 [18:31:46.394]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.439]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:46 [18:31:46.440]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
18:31:46 [18:31:46.542]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.543]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f06ebeb78c8f5896 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f06ebeb78c8f5896"}
18:31:46 [18:31:46.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeu8pKT/1 with map size 1073741824
18:31:46 [18:31:46.548]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.721]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f06ebeb78c8f5896 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f06ebeb78c8f5896"}
18:31:46 [18:31:46.773]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.774]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:29aaeff38838db78 epochNumber=1
{"provingJobId":"1:BASE_PARITY:29aaeff38838db78"}
18:31:46 [18:31:46.775]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testun4WuC/1 with map size 1073741824
18:31:46 [18:31:46.776]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.797]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:29aaeff38838db78 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:29aaeff38838db78"}
18:31:46 [18:31:46.801]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.801]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:817773b0e3c321da epochNumber=1
{"provingJobId":"1:BASE_PARITY:817773b0e3c321da"}
18:31:46 [18:31:46.803]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvMI3m0/1 with map size 1073741824
18:31:46 [18:31:46.805]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.825]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:817773b0e3c321da type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:817773b0e3c321da"}
18:31:46 [18:31:46.826]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:817773b0e3c321da type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:817773b0e3c321da"}
18:31:46 [18:31:46.839]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.840]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd76d023d25cf228 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd76d023d25cf228"}
18:31:46 [18:31:46.840]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6S8Iaj/1 with map size 1073741824
18:31:46 [18:31:46.843]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.867]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dd76d023d25cf228 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:dd76d023d25cf228"}
18:31:46 [18:31:46.877]
INFO:
prover-client:proving-broker Proving Broker started
18:31:46 [18:31:46.879]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a73b1158a6bae52c epochNumber=1
{"provingJobId":"1:BASE_PARITY:a73b1158a6bae52c"}
18:31:46 [18:31:46.879]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testN6NBUC/1 with map size 1073741824
18:31:46 [18:31:46.882]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.910]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a73b1158a6bae52c type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a73b1158a6bae52c"}
18:31:46 [18:31:46.911]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb63a502a7ff2836 epochNumber=3
{"provingJobId":"1:BASE_PARITY:bb63a502a7ff2836"}
18:31:46 [18:31:46.912]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testN6NBUC/3 with map size 1073741824
18:31:46 [18:31:46.915]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.954]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:46 [18:31:46.954]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
18:31:46 [18:31:46.960]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a73b1158a6bae52c err=test error
{"provingJoId":"1:BASE_PARITY:a73b1158a6bae52c"}
18:31:46 [18:31:46.970]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testccKNAk/1 with map size 1073741824
18:31:46 [18:31:46.971]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:46 [18:31:46.993]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testccKNAk/2 with map size 1073741824
18:31:46 [18:31:46.996]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.016]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.017]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ffc98085cf405d69 settled=false
{"provingJobId":"1:BASE_PARITY:ffc98085cf405d69","status":"pending"}
18:31:47 [18:31:47.019]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:95d667e3996ab489 settled=false
{"provingJobId":"1:BASE_PARITY:95d667e3996ab489","status":"pending"}
18:31:47 [18:31:47.027]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQLEN6g/1 with map size 1073741824
18:31:47 [18:31:47.031]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.055]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQLEN6g/2 with map size 1073741824
18:31:47 [18:31:47.058]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.097]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.099]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2e2413e84b1424dd settled=true
{"provingJobId":"1:BASE_PARITY:2e2413e84b1424dd","status":"fulfilled"}
18:31:47 [18:31:47.100]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:ee26d5262013687e settled=true
{"provingJobId":"2:BASE_PARITY:ee26d5262013687e","status":"fulfilled"}
18:31:47 [18:31:47.105]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyZ99m4/1 with map size 1073741824
18:31:47 [18:31:47.109]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.143]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyZ99m4/2 with map size 1073741824
18:31:47 [18:31:47.147]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.170]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.172]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e52c8d26366c1110 settled=true
{"provingJobId":"1:BASE_PARITY:e52c8d26366c1110","status":"fulfilled"}
18:31:47 [18:31:47.174]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:064bd7ab93e2300f settled=false
{"provingJobId":"1:BASE_PARITY:064bd7ab93e2300f","status":"pending"}
18:31:47 [18:31:47.181]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.182]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:747ef5dc5031cace epochNumber=1
{"provingJobId":"1:BASE_PARITY:747ef5dc5031cace"}
18:31:47 [18:31:47.182]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPjm3ap/1 with map size 1073741824
18:31:47 [18:31:47.184]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.208]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.209]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97eee8b409c3f282 epochNumber=1
{"provingJobId":"1:BASE_PARITY:97eee8b409c3f282"}
18:31:47 [18:31:47.210]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:97eee8b409c3f282: Error: db error: Error: db error
18:31:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:97eee8b409c3f282"}
18:31:47 [18:31:47.213]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.214]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a89bf9f9fff38d38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a89bf9f9fff38d38"}
18:31:47 [18:31:47.214]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testE53Ul8/1 with map size 1073741824
18:31:47 [18:31:47.224]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.248]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a89bf9f9fff38d38 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a89bf9f9fff38d38"}
18:31:47 [18:31:47.249]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a89bf9f9fff38d38 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a89bf9f9fff38d38"}
18:31:47 [18:31:47.264]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c38629c40c0c02a epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c38629c40c0c02a"}
18:31:47 [18:31:47.266]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcvq20e/1 with map size 1073741824
18:31:47 [18:31:47.269]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.295]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0c38629c40c0c02a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0c38629c40c0c02a"}
18:31:47 [18:31:47.296]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0c38629c40c0c02a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0c38629c40c0c02a"}
18:31:47 [18:31:47.297]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:0c38629c40c0c02a: Error: db error
18:31:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:0c38629c40c0c02a"}
18:31:47 [18:31:47.302]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.303]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00d43a71f81e3b4d epochNumber=1
{"provingJobId":"1:BASE_PARITY:00d43a71f81e3b4d"}
18:31:47 [18:31:47.305]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH6Nm57/1 with map size 1073741824
18:31:47 [18:31:47.307]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.342]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:00d43a71f81e3b4d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:00d43a71f81e3b4d"}
18:31:47 [18:31:47.343]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:00d43a71f81e3b4d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:00d43a71f81e3b4d"}
18:31:47 [18:31:47.358]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.359]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b7a7b5dea45395e epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b7a7b5dea45395e"}
18:31:47 [18:31:47.361]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtYOtpH/1 with map size 1073741824
18:31:47 [18:31:47.365]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.386]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2b7a7b5dea45395e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2b7a7b5dea45395e"}
18:31:47 [18:31:47.387]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2b7a7b5dea45395e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2b7a7b5dea45395e"}
18:31:47 [18:31:47.388]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:2b7a7b5dea45395e jobErr=test error: Error: db error
18:31:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:2b7a7b5dea45395e"}
18:31:47 [18:31:47.394]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.395]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:76a9922d34d2e2d5 not found
{"provingJobId":"1:BASE_PARITY:76a9922d34d2e2d5"}
18:31:47 [18:31:47.399]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.400]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7ba9203331761c3f err=test error
{"provingJoId":"1:BASE_PARITY:7ba9203331761c3f"}
18:31:47 [18:31:47.406]
INFO:
prover-client:proving-broker Proving Broker started
18:31:47 [18:31:47.422]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:200792199e6f8be4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:200792199e6f8be4"}
18:31:47 [18:31:47.427]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1M4D2K/1 with map size 1073741824
18:31:47 [18:31:47.435]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.453]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:200792199e6f8be4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:200792199e6f8be4"}
18:31:47 [18:31:47.453]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:200792199e6f8be4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:200792199e6f8be4"}
18:31:47 [18:31:47.471]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:3bd970dd09d26d92 epochNumber=2
{"provingJobId":"2:BASE_PARITY:3bd970dd09d26d92"}
18:31:47 [18:31:47.472]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test1M4D2K/2 with map size 1073741824
18:31:47 [18:31:47.478]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.501]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:3bd970dd09d26d92 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:3bd970dd09d26d92"}
18:31:47 [18:31:47.502]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:3bd970dd09d26d92 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:3bd970dd09d26d92"}
18:31:47 [18:31:47.571]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:c4662e7aa525c437 epochNumber=3
{"provingJobId":"3:BASE_PARITY:c4662e7aa525c437"}
18:31:47 [18:31:47.573]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test1M4D2K/3 with map size 1073741824
18:31:47 [18:31:47.576]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.612]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:47 [18:31:47.613]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
18:31:47 [18:31:47.655]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:2a4c334f592c97c0 epochNumber=4
{"provingJobId":"4:BASE_PARITY:2a4c334f592c97c0"}
18:31:47 [18:31:47.656]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-test1M4D2K/4 with map size 1073741824
18:31:47 [18:31:47.661]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.691]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:47 [18:31:47.692]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
18:31:47 [18:31:47.744]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:d178f1f32a430a97 epochNumber=5
{"provingJobId":"5:BASE_PARITY:d178f1f32a430a97"}
18:31:47 [18:31:47.745]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-test1M4D2K/5 with map size 1073741824
18:31:47 [18:31:47.748]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
18:31:47 [18:31:47.771]
INFO:
prover-client:proving-broker Cleaned up jobs=1
18:31:47 [18:31:47.771]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
18:31:47
PASS src/proving_broker/proving_broker.test.ts (
13.267 s)
18:31:47 ProvingBroker
18:31:47 Producer API
18:31:47
✓ refuses stale jobs (52 ms)
18:31:47
✓ enqueues jobs (3 ms)
18:31:47
✓ ignores duplicate jobs (3 ms)
18:31:47
✓ reports correct status when enqueuing repeat jobs (4 ms)
18:31:47
✓ throws an error in case of duplicate job IDs (27 ms)
18:31:47
✓ returns not-found status for non-existing jobs (2 ms)
18:31:47
✓ cancels jobs in queue (6 ms)
18:31:47
✓ cancels jobs in-progress (4 ms)
18:31:47
✓ returns job result if successful (5 ms)
18:31:47
✓ returns job error if failed (5 ms)
18:31:47
✓ correctly returns job status for concurrent writes (3 ms)
18:31:47
✓ refuses stale jobs (40 ms)
18:31:47
✓ enqueues jobs (28 ms)
18:31:47
✓ ignores duplicate jobs (28 ms)
18:31:47
✓ reports correct status when enqueuing repeat jobs (46 ms)
18:31:47
✓ throws an error in case of duplicate job IDs (29 ms)
18:31:47
✓ returns not-found status for non-existing jobs (3 ms)
18:31:47
✓ cancels jobs in queue (33 ms)
18:31:47
✓ cancels jobs in-progress (33 ms)
18:31:47
✓ returns job result if successful (36 ms)
18:31:47
✓ returns job error if failed (34 ms)
18:31:47
✓ correctly returns job status for concurrent writes (38 ms)
18:31:47 Consumer API
18:31:47
✓ returns undefined if no jobs are available (11 ms)
18:31:47
✓ returns jobs in priority order (3 ms)
18:31:47
✓ returns undefined if no jobs are available for the given allowList (1 ms)
18:31:47
✓ returns a job if it is in the allowList (3 ms)
18:31:47
✓ returns the most important job if it is in the allowList (2 ms)
18:31:47
✓ returns any job if filter is empty (2 ms)
18:31:47
✓ returns a new job when reporting job success (2 ms)
18:31:47
✓ returns a new job when reporting permanent error (2 ms)
18:31:47
✓ returns a new job when reporting retry-able error (2 ms)
18:31:47
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
18:31:47
✓ returns a new job if job is already in progress elsewhere (1057 ms)
18:31:47
✓ avoids sending the same job to a new agent after a restart (1006 ms)
18:31:47
✓ avoids sending a completed job to a new agent after a restart (1004 ms)
18:31:47
✓ tracks job result if in progress (3 ms)
18:31:47
✓ tracks job result even if job is in queue (2 ms)
18:31:47
✓ ignores reported job error if unknown job (1 ms)
18:31:47
✓ ignores job result if unknown job (1 ms)
18:31:47
✓ returns undefined if no jobs are available (2 ms)
18:31:47
✓ returns jobs in priority order (49 ms)
18:31:47
✓ returns undefined if no jobs are available for the given allowList (24 ms)
18:31:47
✓ returns a job if it is in the allowList (59 ms)
18:31:47
✓ returns the most important job if it is in the allowList (68 ms)
18:31:47
✓ returns any job if filter is empty (66 ms)
18:31:47
✓ returns a new job when reporting job success (39 ms)
18:31:47
✓ returns a new job when reporting permanent error (43 ms)
18:31:47
✓ returns a new job when reporting retry-able error (37 ms)
18:31:47
✓ returns a new job when reporting progress if current one is cancelled (42 ms)
18:31:47
✓ returns a new job if job is already in progress elsewhere (1129 ms)
18:31:47
✓ avoids sending the same job to a new agent after a restart (1059 ms)
18:31:47
✓ avoids sending a completed job to a new agent after a restart (1086 ms)
18:31:47
✓ tracks job result if in progress (75 ms)
18:31:47
✓ tracks job result even if job is in queue (91 ms)
18:31:47
✓ ignores reported job error if unknown job (5 ms)
18:31:47
✓ ignores job result if unknown job (5 ms)
18:31:47 Timeouts
18:31:47
✓ tracks in progress jobs (1 ms)
18:31:47
✓ re-enqueues jobs that time out (108 ms)
18:31:47
✓ cancel stale jobs that time out (230 ms)
18:31:47
✓ keeps the jobs in progress while it is alive (208 ms)
18:31:47
✓ tracks in progress jobs (28 ms)
18:31:47
✓ re-enqueues jobs that time out (135 ms)
18:31:47
✓ cancel stale jobs that time out (313 ms)
18:31:47
✓ keeps the jobs in progress while it is alive (229 ms)
18:31:47 Retries
18:31:47
✓ retries jobs (6 ms)
18:31:47
✓ retries up to a maximum number of times (4 ms)
18:31:47
✓ passing retry=false does not retry (2 ms)
18:31:47
✓ does not retry if job is stale (29 ms)
18:31:47
✓ retries jobs (27 ms)
18:31:47
✓ retries up to a maximum number of times (37 ms)
18:31:47
✓ passing retry=false does not retry (38 ms)
18:31:47
✓ does not retry if job is stale (89 ms)
18:31:47 Database management
18:31:47
✓ re-enqueues proof requests on start (3 ms)
18:31:47
✓ restores proof results on start (1 ms)
18:31:47
✓ only re-enqueues unfinished jobs (1 ms)
18:31:47
✓ saves job when enqueued (3 ms)
18:31:47
✓ does not retain job if database fails to save (2 ms)
18:31:47
✓ saves job result (2 ms)
18:31:47
✓ saves result even if database fails to save (3 ms)
18:31:47
✓ saves job error (2 ms)
18:31:47
✓ saves job error even if database fails to save (2 ms)
18:31:47
✓ does not save job result if job is unknown (1 ms)
18:31:47
✓ does not save job error if job is unknown (1 ms)
18:31:47
✓ cleans up old jobs periodically (260 ms)
18:31:47
✓ re-enqueues proof requests on start (58 ms)
18:31:47
✓ restores proof results on start (80 ms)
18:31:47
✓ only re-enqueues unfinished jobs (76 ms)
18:31:47
✓ saves job when enqueued (25 ms)
18:31:47
✓ does not retain job if database fails to save (5 ms)
18:31:47
✓ saves job result (50 ms)
18:31:47
✓ saves result even if database fails to save (38 ms)
18:31:47
✓ saves job error (54 ms)
18:31:47
✓ saves job error even if database fails to save (36 ms)
18:31:47
✓ does not save job result if job is unknown (4 ms)
18:31:47
✓ does not save job error if job is unknown (5 ms)
18:31:47
✓ cleans up old jobs periodically (421 ms)
18:31:47
18:31:47
Test Suites: 1 passed, 1 total
18:31:47
Tests: 96 passed, 96 total
18:31:47
Snapshots: 0 total
18:31:47
Time: 13.375 s
18:31:47
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
18:31:47
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?