Parent Log:
http://ci.aztec-labs.com/d5c4976d91abf392
Command: 72b59a67627043b2 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/433ba6218aed45169029991a493e4232b30b8ec6
Env: REF_NAME=gh-readonly-queue/next/pr-15114-23ff13c6b16a0c1333bf26aa5c213054c92403c3 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 17 17:53:17 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15114_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
17:53:21 [17:53:21.685]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.691]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53d5ff33371fb9e7 epochNumber=42
{"provingJobId":"1:BASE_PARITY:53d5ff33371fb9e7"}
17:53:21 [17:53:21.694]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:2a7b7f4311f7bb3d epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a7b7f4311f7bb3d"}
17:53:21 [17:53:21.720]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.721]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44d43f991014c330 epochNumber=1
{"provingJobId":"1:BASE_PARITY:44d43f991014c330"}
17:53:21 [17:53:21.722]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2684b773a0d8179 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2684b773a0d8179"}
17:53:21 [17:53:21.723]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.724]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fdeca1941229b53b epochNumber=1
{"provingJobId":"1:BASE_PARITY:fdeca1941229b53b"}
17:53:21 [17:53:21.725]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fdeca1941229b53b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fdeca1941229b53b"}
17:53:21 [17:53:21.727]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.728]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:821aa579126aa31e epochNumber=1
{"provingJobId":"1:BASE_PARITY:821aa579126aa31e"}
17:53:21 [17:53:21.728]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:821aa579126aa31e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:821aa579126aa31e"}
17:53:21 [17:53:21.729]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:821aa579126aa31e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:821aa579126aa31e"}
17:53:21 [17:53:21.729]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:821aa579126aa31e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:821aa579126aa31e"}
17:53:21 [17:53:21.730]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.731]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:842ee8dcc761307a epochNumber=1
{"provingJobId":"1:BASE_PARITY:842ee8dcc761307a"}
17:53:21 [17:53:21.755]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.756]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.757]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1821313c2dbe16b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1821313c2dbe16b0"}
17:53:21 [17:53:21.757]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1821313c2dbe16b0
{"provingJobId":"1:BASE_PARITY:1821313c2dbe16b0"}
17:53:21 [17:53:21.758]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1821313c2dbe16b0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1821313c2dbe16b0"}
17:53:21 [17:53:21.758]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1821313c2dbe16b0 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1821313c2dbe16b0"}
17:53:21 [17:53:21.759]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.760]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d186f85bba648c5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:d186f85bba648c5d"}
17:53:21 [17:53:21.760]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d186f85bba648c5d
{"provingJobId":"1:BASE_PARITY:d186f85bba648c5d"}
17:53:21 [17:53:21.761]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d186f85bba648c5d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:d186f85bba648c5d"}
17:53:21 [17:53:21.762]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.763]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3471562820bb9be1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3471562820bb9be1"}
17:53:21 [17:53:21.763]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3471562820bb9be1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3471562820bb9be1"}
17:53:21 [17:53:21.764]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3471562820bb9be1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3471562820bb9be1"}
17:53:21 [17:53:21.765]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.766]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4ad994bef2b7cb3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4ad994bef2b7cb3"}
17:53:21 [17:53:21.766]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d4ad994bef2b7cb3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d4ad994bef2b7cb3"}
17:53:21 [17:53:21.766]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d4ad994bef2b7cb3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d4ad994bef2b7cb3"}
17:53:21 [17:53:21.767]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.768]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3692d911bb3b5120 epochNumber=0
{"provingJobId":"1:BASE_PARITY:3692d911bb3b5120"}
17:53:21 [17:53:21.768]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3692d911bb3b5120 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3692d911bb3b5120"}
17:53:21 [17:53:21.768]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fbae7e36594df990 epochNumber=0
{"provingJobId":"1:BASE_PARITY:fbae7e36594df990"}
17:53:21 [17:53:21.769]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3692d911bb3b5120 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3692d911bb3b5120"}
17:53:21 [17:53:21.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:23dfb7b8615da72e epochNumber=0
{"provingJobId":"1:BASE_PARITY:23dfb7b8615da72e"}
17:53:21 [17:53:21.770]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.771]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.771]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df598222faca94d9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:df598222faca94d9"}
17:53:21 [17:53:21.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:848e358a8f2f4060 epochNumber=1
{"provingJobId":"1:BASE_PARITY:848e358a8f2f4060"}
17:53:21 [17:53:21.773]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.773]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6493aed7555f5b0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:6493aed7555f5b0b"}
17:53:21 [17:53:21.781]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60dbaeefa0ce511d epochNumber=1
{"provingJobId":"1:BASE_PARITY:60dbaeefa0ce511d"}
17:53:21 [17:53:21.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c19e0ca3792ddaa epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c19e0ca3792ddaa"}
17:53:21 [17:53:21.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75f6f48249c2607e epochNumber=2
{"provingJobId":"1:BASE_PARITY:75f6f48249c2607e"}
17:53:21 [17:53:21.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cbbbd2718903c6f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cbbbd2718903c6f8"}
17:53:21 [17:53:21.784]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bdb82a84c1ae230f epochNumber=1
{"provingJobId":"1:BASE_PARITY:bdb82a84c1ae230f"}
17:53:21 [17:53:21.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:628b193af92a4bc7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:628b193af92a4bc7"}
17:53:21 [17:53:21.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f59844299ced862e epochNumber=2
{"provingJobId":"1:BASE_PARITY:f59844299ced862e"}
17:53:21 [17:53:21.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5eba34805c8a0270 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5eba34805c8a0270"}
17:53:21 [17:53:21.787]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.787]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35b8b03264b7f12e epochNumber=1
{"provingJobId":"1:BASE_PARITY:35b8b03264b7f12e"}
17:53:21 [17:53:21.787]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88dcfb068b3955cc epochNumber=1
{"provingJobId":"1:BASE_PARITY:88dcfb068b3955cc"}
17:53:21 [17:53:21.788]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06ca1c94f9e19c2c epochNumber=2
{"provingJobId":"1:BASE_PARITY:06ca1c94f9e19c2c"}
17:53:21 [17:53:21.788]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ae3e8b0e8bf0333 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ae3e8b0e8bf0333"}
17:53:21 [17:53:21.789]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.790]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8697b4d1625828d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8697b4d1625828d0"}
17:53:21 [17:53:21.790]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa7bc4b1dc704624 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa7bc4b1dc704624"}
17:53:21 [17:53:21.790]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8697b4d1625828d0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8697b4d1625828d0"}
17:53:21 [17:53:21.792]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.792]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:347dd5dd244a17ef epochNumber=1
{"provingJobId":"1:BASE_PARITY:347dd5dd244a17ef"}
17:53:21 [17:53:21.792]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:344e0af96ac77a81 epochNumber=1
{"provingJobId":"1:BASE_PARITY:344e0af96ac77a81"}
17:53:21 [17:53:21.793]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:347dd5dd244a17ef type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:347dd5dd244a17ef"}
17:53:21 [17:53:21.794]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.794]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cfd47c0db15ba3fc epochNumber=1
{"provingJobId":"1:BASE_PARITY:cfd47c0db15ba3fc"}
17:53:21 [17:53:21.795]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b87a6dd89f6417c epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b87a6dd89f6417c"}
17:53:21 [17:53:21.795]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cfd47c0db15ba3fc type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:cfd47c0db15ba3fc"}
17:53:21 [17:53:21.796]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.796]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46f9a9e91a26da9a epochNumber=1
{"provingJobId":"1:BASE_PARITY:46f9a9e91a26da9a"}
17:53:21 [17:53:21.797]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:46f9a9e91a26da9a
{"provingJobId":"1:BASE_PARITY:46f9a9e91a26da9a"}
17:53:21 [17:53:21.797]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:46f9a9e91a26da9a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:46f9a9e91a26da9a"}
17:53:21 [17:53:21.797]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c3129a5fcf23eb19 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c3129a5fcf23eb19"}
17:53:21 [17:53:21.798]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:46f9a9e91a26da9a has already been completed
{"provingJobId":"1:BASE_PARITY:46f9a9e91a26da9a"}
17:53:21 [17:53:21.799]
INFO:
prover-client:proving-broker Proving Broker started
17:53:21 [17:53:21.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6bcbbe98280b3c47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6bcbbe98280b3c47"}
17:53:21 [17:53:21.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64fba2a74121ffe8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:64fba2a74121ffe8"}
17:53:22 [17:53:22.851]
INFO:
prover-client:proving-broker Proving Broker started
17:53:22 [17:53:22.852]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6bcbbe98280b3c47 settled=false
{"provingJobId":"1:BASE_PARITY:6bcbbe98280b3c47","status":"pending"}
17:53:22 [17:53:22.853]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:64fba2a74121ffe8 settled=false
{"provingJobId":"1:BASE_PARITY:64fba2a74121ffe8","status":"pending"}
17:53:22 [17:53:22.854]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6bcbbe98280b3c47 type=BASE_PARITY startedAt=1750182801800 older agent has taken job
{"provingJobId":"1:BASE_PARITY:6bcbbe98280b3c47"}
17:53:22 [17:53:22.854]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6bcbbe98280b3c47 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:6bcbbe98280b3c47"}
17:53:22 [17:53:22.855]
INFO:
prover-client:proving-broker Proving Broker started
17:53:22 [17:53:22.856]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e14c68c7da14a72b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e14c68c7da14a72b"}
17:53:22 [17:53:22.856]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a869bbc3bde63833 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a869bbc3bde63833"}
17:53:23 [17:53:23.857]
INFO:
prover-client:proving-broker Proving Broker started
17:53:23 [17:53:23.858]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e14c68c7da14a72b settled=false
{"provingJobId":"1:BASE_PARITY:e14c68c7da14a72b","status":"pending"}
17:53:23 [17:53:23.858]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a869bbc3bde63833 settled=false
{"provingJobId":"1:BASE_PARITY:a869bbc3bde63833","status":"pending"}
17:53:23 [17:53:23.859]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e14c68c7da14a72b type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:e14c68c7da14a72b"}
17:53:23 [17:53:23.861]
INFO:
prover-client:proving-broker Proving Broker started
17:53:23 [17:53:23.861]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:589029acf6af90f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:589029acf6af90f5"}
17:53:23 [17:53:23.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3009cdeb3941fd3f epochNumber=2
{"provingJobId":"1:BASE_PARITY:3009cdeb3941fd3f"}
17:53:24 [17:53:24.863]
INFO:
prover-client:proving-broker Proving Broker started
17:53:24 [17:53:24.864]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:589029acf6af90f5 settled=false
{"provingJobId":"1:BASE_PARITY:589029acf6af90f5","status":"pending"}
17:53:24 [17:53:24.864]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3009cdeb3941fd3f settled=false
{"provingJobId":"1:BASE_PARITY:3009cdeb3941fd3f","status":"pending"}
17:53:24 [17:53:24.865]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:589029acf6af90f5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:589029acf6af90f5"}
17:53:24 [17:53:24.865]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:589029acf6af90f5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:589029acf6af90f5"}
17:53:24 [17:53:24.866]
INFO:
prover-client:proving-broker Proving Broker started
17:53:24 [17:53:24.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb4eb7a1573731f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb4eb7a1573731f5"}
17:53:24 [17:53:24.874]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f343ada5efad38f epochNumber=2
{"provingJobId":"1:BASE_PARITY:4f343ada5efad38f"}
17:53:24 [17:53:24.874]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cb4eb7a1573731f5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cb4eb7a1573731f5"}
17:53:24 [17:53:24.875]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4f343ada5efad38f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4f343ada5efad38f"}
17:53:24 [17:53:24.876]
INFO:
prover-client:proving-broker Proving Broker started
17:53:24 [17:53:24.876]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21a3afd0312ad87c epochNumber=1
{"provingJobId":"1:BASE_PARITY:21a3afd0312ad87c"}
17:53:24 [17:53:24.876]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b98afa572b8e2d15 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b98afa572b8e2d15"}
17:53:24 [17:53:24.876]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:21a3afd0312ad87c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:21a3afd0312ad87c"}
17:53:24 [17:53:24.877]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:21a3afd0312ad87c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:21a3afd0312ad87c"}
17:53:24 [17:53:24.877]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b98afa572b8e2d15 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b98afa572b8e2d15"}
17:53:24 [17:53:24.877]
INFO:
prover-client:proving-broker Proving Broker started
17:53:24 [17:53:24.878]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ac7d3cb3b90d362c err=test error
{"provingJoId":"1:BASE_PARITY:ac7d3cb3b90d362c"}
17:53:24 [17:53:24.879]
INFO:
prover-client:proving-broker Proving Broker started
17:53:24 [17:53:24.879]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:63c66f5e3116b048 not found
{"provingJobId":"1:BASE_PARITY:63c66f5e3116b048"}
17:53:24 [17:53:24.880]
INFO:
prover-client:proving-broker Proving Broker started
17:53:24 [17:53:24.881]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd802a13ba6944ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd802a13ba6944ca"}
17:53:24 [17:53:24.882]
INFO:
prover-client:proving-broker Proving Broker started
17:53:24 [17:53:24.882]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da9b2a48d7415e97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:da9b2a48d7415e97"}
17:53:25 [17:53:25.007]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:da9b2a48d7415e97 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:da9b2a48d7415e97"}
17:53:25 [17:53:25.008]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.009]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1da0f511e783d2f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1da0f511e783d2f"}
17:53:25 [17:53:25.110]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a1da0f511e783d2f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a1da0f511e783d2f"}
17:53:25 [17:53:25.117]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01ac0cdfdfac18e0 epochNumber=10
{"provingJobId":"1:BASE_PARITY:01ac0cdfdfac18e0"}
17:53:25 [17:53:25.137]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:25 [17:53:25.242]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.243]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1778e07c3adced8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1778e07c3adced8"}
17:53:25 [17:53:25.396]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e1778e07c3adced8 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e1778e07c3adced8"}
17:53:25 [17:53:25.446]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8b85a79f3530423 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f8b85a79f3530423"}
17:53:25 [17:53:25.448]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f8b85a79f3530423 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f8b85a79f3530423"}
17:53:25 [17:53:25.449]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.449]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9946eea683a8132 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9946eea683a8132"}
17:53:25 [17:53:25.450]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e9946eea683a8132 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:e9946eea683a8132"}
17:53:25 [17:53:25.451]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e9946eea683a8132 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:e9946eea683a8132"}
17:53:25 [17:53:25.452]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:241621d0d6360202 epochNumber=1
{"provingJobId":"1:BASE_PARITY:241621d0d6360202"}
17:53:25 [17:53:25.453]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:241621d0d6360202 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:241621d0d6360202"}
17:53:25 [17:53:25.454]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.455]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6fea019b669a0213 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6fea019b669a0213"}
17:53:25 [17:53:25.455]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6fea019b669a0213 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6fea019b669a0213"}
17:53:25 [17:53:25.456]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb66e78650c986b1 epochNumber=3
{"provingJobId":"1:BASE_PARITY:cb66e78650c986b1"}
17:53:25 [17:53:25.480]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:25 [17:53:25.481]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6fea019b669a0213 err=test error
{"provingJoId":"1:BASE_PARITY:6fea019b669a0213"}
17:53:25 [17:53:25.483]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.483]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4749a8fa407389aa settled=false
{"provingJobId":"1:BASE_PARITY:4749a8fa407389aa","status":"pending"}
17:53:25 [17:53:25.483]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c536c69aecdc5d51 settled=false
{"provingJobId":"1:BASE_PARITY:c536c69aecdc5d51","status":"pending"}
17:53:25 [17:53:25.487]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.487]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c46e2b470ad9330f settled=true
{"provingJobId":"1:BASE_PARITY:c46e2b470ad9330f","status":"fulfilled"}
17:53:25 [17:53:25.487]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:ef1db6214f3e6dc9 settled=true
{"provingJobId":"2:BASE_PARITY:ef1db6214f3e6dc9","status":"fulfilled"}
17:53:25 [17:53:25.489]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.489]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:75ac97fa53556769 settled=true
{"provingJobId":"1:BASE_PARITY:75ac97fa53556769","status":"fulfilled"}
17:53:25 [17:53:25.490]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d566dded5abfe638 settled=false
{"provingJobId":"1:BASE_PARITY:d566dded5abfe638","status":"pending"}
17:53:25 [17:53:25.491]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.493]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd8347cf455e99a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd8347cf455e99a4"}
17:53:25 [17:53:25.494]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.495]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08a9272c7c048d51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:08a9272c7c048d51"}
17:53:25 [17:53:25.496]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:08a9272c7c048d51: Error: db error: Error: db error
17:53:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:08a9272c7c048d51"}
17:53:25 [17:53:25.497]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.498]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ce816e0d462e0ce epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ce816e0d462e0ce"}
17:53:25 [17:53:25.499]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7ce816e0d462e0ce type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7ce816e0d462e0ce"}
17:53:25 [17:53:25.500]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7ce816e0d462e0ce type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7ce816e0d462e0ce"}
17:53:25 [17:53:25.502]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.503]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1540880ef8735fd8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1540880ef8735fd8"}
17:53:25 [17:53:25.503]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1540880ef8735fd8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1540880ef8735fd8"}
17:53:25 [17:53:25.503]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1540880ef8735fd8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1540880ef8735fd8"}
17:53:25 [17:53:25.504]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:1540880ef8735fd8: Error: db error
17:53:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:1540880ef8735fd8"}
17:53:25 [17:53:25.505]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.507]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3076438f9b15d54c epochNumber=1
{"provingJobId":"1:BASE_PARITY:3076438f9b15d54c"}
17:53:25 [17:53:25.507]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3076438f9b15d54c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3076438f9b15d54c"}
17:53:25 [17:53:25.508]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3076438f9b15d54c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3076438f9b15d54c"}
17:53:25 [17:53:25.510]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92c687b295fb2516 epochNumber=1
{"provingJobId":"1:BASE_PARITY:92c687b295fb2516"}
17:53:25 [17:53:25.511]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:92c687b295fb2516 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:92c687b295fb2516"}
17:53:25 [17:53:25.511]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:92c687b295fb2516 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:92c687b295fb2516"}
17:53:25 [17:53:25.512]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:92c687b295fb2516 jobErr=test error: Error: db error
17:53:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:92c687b295fb2516"}
17:53:25 [17:53:25.513]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.513]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3c8a9844f4d5a2d8 not found
{"provingJobId":"1:BASE_PARITY:3c8a9844f4d5a2d8"}
17:53:25 [17:53:25.519]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.520]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:92e6ec4aab35e7d5 err=test error
{"provingJoId":"1:BASE_PARITY:92e6ec4aab35e7d5"}
17:53:25 [17:53:25.522]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.532]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9eadbbdc7f055bd7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9eadbbdc7f055bd7"}
17:53:25 [17:53:25.533]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9eadbbdc7f055bd7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9eadbbdc7f055bd7"}
17:53:25 [17:53:25.534]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9eadbbdc7f055bd7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9eadbbdc7f055bd7"}
17:53:25 [17:53:25.544]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:f99b017196c623a2 epochNumber=2
{"provingJobId":"2:BASE_PARITY:f99b017196c623a2"}
17:53:25 [17:53:25.545]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:f99b017196c623a2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:f99b017196c623a2"}
17:53:25 [17:53:25.545]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:f99b017196c623a2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:f99b017196c623a2"}
17:53:25 [17:53:25.606]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:fc1318e0bf647c3a epochNumber=3
{"provingJobId":"3:BASE_PARITY:fc1318e0bf647c3a"}
17:53:25 [17:53:25.625]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:25 [17:53:25.668]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:57a6bd8666f28581 epochNumber=4
{"provingJobId":"4:BASE_PARITY:57a6bd8666f28581"}
17:53:25 [17:53:25.675]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:25 [17:53:25.731]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:d8191e0a7b3173cf epochNumber=5
{"provingJobId":"5:BASE_PARITY:d8191e0a7b3173cf"}
17:53:25 [17:53:25.750]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:25 [17:53:25.785]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e156a0be8a665830 epochNumber=42
{"provingJobId":"1:BASE_PARITY:e156a0be8a665830"}
17:53:25 [17:53:25.787]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testVkRJ9R/42 with map size 1073741824
17:53:25 [17:53:25.790]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:25 [17:53:25.816]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:f0b03a5ec1c47f16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f0b03a5ec1c47f16"}
17:53:25 [17:53:25.822]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.823]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ab75249dae9dad2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ab75249dae9dad2"}
17:53:25 [17:53:25.824]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpTXOzC/1 with map size 1073741824
17:53:25 [17:53:25.826]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:25 [17:53:25.847]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c49894dea9243a42 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c49894dea9243a42"}
17:53:25 [17:53:25.857]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.858]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0294d9bbf68bc148 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0294d9bbf68bc148"}
17:53:25 [17:53:25.859]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiWTQB3/1 with map size 1073741824
17:53:25 [17:53:25.861]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:25 [17:53:25.879]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0294d9bbf68bc148 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0294d9bbf68bc148"}
17:53:25 [17:53:25.884]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.885]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1df87a2d9f91f0f1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1df87a2d9f91f0f1"}
17:53:25 [17:53:25.886]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdg2uqd/1 with map size 1073741824
17:53:25 [17:53:25.888]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:25 [17:53:25.905]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1df87a2d9f91f0f1 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1df87a2d9f91f0f1"}
17:53:25 [17:53:25.906]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1df87a2d9f91f0f1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1df87a2d9f91f0f1"}
17:53:25 [17:53:25.912]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1df87a2d9f91f0f1 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1df87a2d9f91f0f1"}
17:53:25 [17:53:25.916]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.917]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c058705b6951c04 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c058705b6951c04"}
17:53:25 [17:53:25.918]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPPdcak/1 with map size 1073741824
17:53:25 [17:53:25.919]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:25 [17:53:25.940]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.943]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c27f29224df40a26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c27f29224df40a26"}
17:53:25 [17:53:25.944]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtFtmLY/1 with map size 1073741824
17:53:25 [17:53:25.946]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:25 [17:53:25.963]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c27f29224df40a26
{"provingJobId":"1:BASE_PARITY:c27f29224df40a26"}
17:53:25 [17:53:25.963]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c27f29224df40a26 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c27f29224df40a26"}
17:53:25 [17:53:25.963]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c27f29224df40a26 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c27f29224df40a26"}
17:53:25 [17:53:25.973]
INFO:
prover-client:proving-broker Proving Broker started
17:53:25 [17:53:25.973]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a2ec82806f1d3df epochNumber=1
{"provingJobId":"1:BASE_PARITY:6a2ec82806f1d3df"}
17:53:25 [17:53:25.974]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEwEAZu/1 with map size 1073741824
17:53:25 [17:53:25.975]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:25 [17:53:25.993]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6a2ec82806f1d3df
{"provingJobId":"1:BASE_PARITY:6a2ec82806f1d3df"}
17:53:25 [17:53:25.993]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6a2ec82806f1d3df type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:6a2ec82806f1d3df"}
17:53:26 [17:53:26.003]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.004]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b681118f75d9a8a epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b681118f75d9a8a"}
17:53:26 [17:53:26.004]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLf6Bit/1 with map size 1073741824
17:53:26 [17:53:26.006]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.021]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9b681118f75d9a8a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9b681118f75d9a8a"}
17:53:26 [17:53:26.022]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9b681118f75d9a8a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9b681118f75d9a8a"}
17:53:26 [17:53:26.029]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.030]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b5e8a7c77388472 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b5e8a7c77388472"}
17:53:26 [17:53:26.030]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCXmFvX/1 with map size 1073741824
17:53:26 [17:53:26.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.049]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2b5e8a7c77388472 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2b5e8a7c77388472"}
17:53:26 [17:53:26.049]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2b5e8a7c77388472 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2b5e8a7c77388472"}
17:53:26 [17:53:26.058]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.058]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19a2f9538c7f2ac9 epochNumber=0
{"provingJobId":"1:BASE_PARITY:19a2f9538c7f2ac9"}
17:53:26 [17:53:26.059]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testEIe8lo/0 with map size 1073741824
17:53:26 [17:53:26.060]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.081]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:19a2f9538c7f2ac9 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:19a2f9538c7f2ac9"}
17:53:26 [17:53:26.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:29f121a81d17b21c epochNumber=0
{"provingJobId":"1:BASE_PARITY:29f121a81d17b21c"}
17:53:26 [17:53:26.082]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:19a2f9538c7f2ac9 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:19a2f9538c7f2ac9"}
17:53:26 [17:53:26.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d1e686558186ef0 epochNumber=0
{"provingJobId":"1:BASE_PARITY:8d1e686558186ef0"}
17:53:26 [17:53:26.097]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.101]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.101]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e59d69340398966 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0e59d69340398966"}
17:53:26 [17:53:26.102]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testaH0IhM/2 with map size 1073741824
17:53:26 [17:53:26.104]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.125]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1cd5124c885b47c epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1cd5124c885b47c"}
17:53:26 [17:53:26.126]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaH0IhM/1 with map size 1073741824
17:53:26 [17:53:26.129]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.154]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.156]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a41d6987fae3fb62 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a41d6987fae3fb62"}
17:53:26 [17:53:26.156]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testx8ZSco/1 with map size 1073741824
17:53:26 [17:53:26.160]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.181]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.182]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bec69db67508f47a epochNumber=1
{"provingJobId":"1:BASE_PARITY:bec69db67508f47a"}
17:53:26 [17:53:26.184]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testX9fj1r/1 with map size 1073741824
17:53:26 [17:53:26.187]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.205]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:882702b17d96ad3c epochNumber=1
{"provingJobId":"1:BASE_PARITY:882702b17d96ad3c"}
17:53:26 [17:53:26.211]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96b2d691bd0ca7a6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:96b2d691bd0ca7a6"}
17:53:26 [17:53:26.212]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testX9fj1r/2 with map size 1073741824
17:53:26 [17:53:26.214]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:104bbc0a2b6cbb50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:104bbc0a2b6cbb50"}
17:53:26 [17:53:26.243]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.245]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b69b6da1ebea9eac epochNumber=1
{"provingJobId":"1:BASE_PARITY:b69b6da1ebea9eac"}
17:53:26 [17:53:26.245]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPDGCTH/1 with map size 1073741824
17:53:26 [17:53:26.248]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.267]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dba9623a22d56179 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dba9623a22d56179"}
17:53:26 [17:53:26.275]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6946f1881f3f17b epochNumber=2
{"provingJobId":"1:BASE_PARITY:c6946f1881f3f17b"}
17:53:26 [17:53:26.276]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testPDGCTH/2 with map size 1073741824
17:53:26 [17:53:26.278]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.296]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c9dd5648a4a26b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c9dd5648a4a26b6"}
17:53:26 [17:53:26.307]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.308]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f94edc7bceb85ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f94edc7bceb85ad"}
17:53:26 [17:53:26.309]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFfaWg4/1 with map size 1073741824
17:53:26 [17:53:26.310]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.327]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7fd581195b6864f epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7fd581195b6864f"}
17:53:26 [17:53:26.333]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a5561d5fa3d1c2e2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a5561d5fa3d1c2e2"}
17:53:26 [17:53:26.334]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFfaWg4/2 with map size 1073741824
17:53:26 [17:53:26.336]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.353]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96d259a40e7dae35 epochNumber=1
{"provingJobId":"1:BASE_PARITY:96d259a40e7dae35"}
17:53:26 [17:53:26.362]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.364]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5756883a0ab49bc9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5756883a0ab49bc9"}
17:53:26 [17:53:26.364]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7CwNaQ/1 with map size 1073741824
17:53:26 [17:53:26.366]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.385]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d0dcc7dd6cfcde7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1d0dcc7dd6cfcde7"}
17:53:26 [17:53:26.391]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5756883a0ab49bc9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5756883a0ab49bc9"}
17:53:26 [17:53:26.403]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.404]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:048e0030f97541b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:048e0030f97541b3"}
17:53:26 [17:53:26.405]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testURKaWc/1 with map size 1073741824
17:53:26 [17:53:26.407]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.426]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a34db1e03a59789 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a34db1e03a59789"}
17:53:26 [17:53:26.433]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:048e0030f97541b3 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:048e0030f97541b3"}
17:53:26 [17:53:26.445]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.446]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a2ecacead43bafd2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a2ecacead43bafd2"}
17:53:26 [17:53:26.451]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWuphy1/1 with map size 1073741824
17:53:26 [17:53:26.459]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:967fe0a4d9f1cc2c epochNumber=1
{"provingJobId":"1:BASE_PARITY:967fe0a4d9f1cc2c"}
17:53:26 [17:53:26.490]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a2ecacead43bafd2 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:a2ecacead43bafd2"}
17:53:26 [17:53:26.496]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.497]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14055130d9605d28 epochNumber=1
{"provingJobId":"1:BASE_PARITY:14055130d9605d28"}
17:53:26 [17:53:26.499]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4aLaKv/1 with map size 1073741824
17:53:26 [17:53:26.503]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.522]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:14055130d9605d28
{"provingJobId":"1:BASE_PARITY:14055130d9605d28"}
17:53:26 [17:53:26.522]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:14055130d9605d28 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:14055130d9605d28"}
17:53:26 [17:53:26.529]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ced778335e4d9c8d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ced778335e4d9c8d"}
17:53:26 [17:53:26.536]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:14055130d9605d28 has already been completed
{"provingJobId":"1:BASE_PARITY:14055130d9605d28"}
17:53:26 [17:53:26.540]
INFO:
prover-client:proving-broker Proving Broker started
17:53:26 [17:53:26.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a75928916edbde28 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a75928916edbde28"}
17:53:26 [17:53:26.541]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBo9wcX/1 with map size 1073741824
17:53:26 [17:53:26.543]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:26 [17:53:26.566]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4d10618735f83b7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f4d10618735f83b7"}
17:53:26 [17:53:26.567]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testBo9wcX/2 with map size 1073741824
17:53:26 [17:53:26.572]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:27 [17:53:27.644]
INFO:
prover-client:proving-broker Proving Broker started
17:53:27 [17:53:27.651]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a75928916edbde28 settled=false
{"provingJobId":"1:BASE_PARITY:a75928916edbde28","status":"pending"}
17:53:27 [17:53:27.653]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f4d10618735f83b7 settled=false
{"provingJobId":"1:BASE_PARITY:f4d10618735f83b7","status":"pending"}
17:53:27 [17:53:27.654]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a75928916edbde28 type=BASE_PARITY startedAt=1750182806590 older agent has taken job
{"provingJobId":"1:BASE_PARITY:a75928916edbde28"}
17:53:27 [17:53:27.654]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a75928916edbde28 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:a75928916edbde28"}
17:53:27 [17:53:27.660]
INFO:
prover-client:proving-broker Proving Broker started
17:53:27 [17:53:27.661]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88decf58c8b01589 epochNumber=1
{"provingJobId":"1:BASE_PARITY:88decf58c8b01589"}
17:53:27 [17:53:27.662]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmQBVYR/1 with map size 1073741824
17:53:27 [17:53:27.664]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:27 [17:53:27.689]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7fba7c78015afcba epochNumber=2
{"provingJobId":"1:BASE_PARITY:7fba7c78015afcba"}
17:53:27 [17:53:27.690]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testmQBVYR/2 with map size 1073741824
17:53:27 [17:53:27.693]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:28 [17:53:28.717]
INFO:
prover-client:proving-broker Proving Broker started
17:53:28 [17:53:28.719]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:88decf58c8b01589 settled=false
{"provingJobId":"1:BASE_PARITY:88decf58c8b01589","status":"pending"}
17:53:28 [17:53:28.721]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7fba7c78015afcba settled=false
{"provingJobId":"1:BASE_PARITY:7fba7c78015afcba","status":"pending"}
17:53:28 [17:53:28.722]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:88decf58c8b01589 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:88decf58c8b01589"}
17:53:28 [17:53:28.725]
INFO:
prover-client:proving-broker Proving Broker started
17:53:28 [17:53:28.726]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5f44c75d52d7d3b epochNumber=1
{"provingJobId":"1:BASE_PARITY:c5f44c75d52d7d3b"}
17:53:28 [17:53:28.726]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testF594ja/1 with map size 1073741824
17:53:28 [17:53:28.728]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:28 [17:53:28.746]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d274930678e3379f epochNumber=2
{"provingJobId":"1:BASE_PARITY:d274930678e3379f"}
17:53:28 [17:53:28.747]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testF594ja/2 with map size 1073741824
17:53:28 [17:53:28.749]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:29 [17:53:29.775]
INFO:
prover-client:proving-broker Proving Broker started
17:53:29 [17:53:29.777]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c5f44c75d52d7d3b settled=false
{"provingJobId":"1:BASE_PARITY:c5f44c75d52d7d3b","status":"pending"}
17:53:29 [17:53:29.779]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d274930678e3379f settled=false
{"provingJobId":"1:BASE_PARITY:d274930678e3379f","status":"pending"}
17:53:29 [17:53:29.780]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c5f44c75d52d7d3b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c5f44c75d52d7d3b"}
17:53:29 [17:53:29.780]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c5f44c75d52d7d3b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c5f44c75d52d7d3b"}
17:53:29 [17:53:29.790]
INFO:
prover-client:proving-broker Proving Broker started
17:53:29 [17:53:29.790]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef9958e0135230fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef9958e0135230fd"}
17:53:29 [17:53:29.791]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGqaq1Q/1 with map size 1073741824
17:53:29 [17:53:29.793]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:29 [17:53:29.809]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ab1e7e43c139409 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4ab1e7e43c139409"}
17:53:29 [17:53:29.810]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGqaq1Q/2 with map size 1073741824
17:53:29 [17:53:29.812]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:29 [17:53:29.830]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ef9958e0135230fd type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ef9958e0135230fd"}
17:53:29 [17:53:29.839]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4ab1e7e43c139409 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4ab1e7e43c139409"}
17:53:29 [17:53:29.849]
INFO:
prover-client:proving-broker Proving Broker started
17:53:29 [17:53:29.850]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5fd6e83abfa723d epochNumber=1
{"provingJobId":"1:BASE_PARITY:f5fd6e83abfa723d"}
17:53:29 [17:53:29.850]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYoUyEm/1 with map size 1073741824
17:53:29 [17:53:29.852]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:29 [17:53:29.870]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5aa47192b53bb0be epochNumber=2
{"provingJobId":"1:BASE_PARITY:5aa47192b53bb0be"}
17:53:29 [17:53:29.871]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYoUyEm/2 with map size 1073741824
17:53:29 [17:53:29.873]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:29 [17:53:29.889]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f5fd6e83abfa723d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f5fd6e83abfa723d"}
17:53:29 [17:53:29.889]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f5fd6e83abfa723d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f5fd6e83abfa723d"}
17:53:29 [17:53:29.896]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5aa47192b53bb0be type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5aa47192b53bb0be"}
17:53:29 [17:53:29.905]
INFO:
prover-client:proving-broker Proving Broker started
17:53:29 [17:53:29.905]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:845fe07202844641 err=test error
{"provingJoId":"1:BASE_PARITY:845fe07202844641"}
17:53:29 [17:53:29.909]
INFO:
prover-client:proving-broker Proving Broker started
17:53:29 [17:53:29.910]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b7a700079049d10c not found
{"provingJobId":"1:BASE_PARITY:b7a700079049d10c"}
17:53:29 [17:53:29.914]
INFO:
prover-client:proving-broker Proving Broker started
17:53:29 [17:53:29.915]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:274d7bb04f3b87cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:274d7bb04f3b87cb"}
17:53:29 [17:53:29.916]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlLGd8y/1 with map size 1073741824
17:53:29 [17:53:29.918]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:29 [17:53:29.938]
INFO:
prover-client:proving-broker Proving Broker started
17:53:29 [17:53:29.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47dc7e8026b1d606 epochNumber=1
{"provingJobId":"1:BASE_PARITY:47dc7e8026b1d606"}
17:53:29 [17:53:29.939]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOX6VaP/1 with map size 1073741824
17:53:29 [17:53:29.941]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.065]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:47dc7e8026b1d606 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:47dc7e8026b1d606"}
17:53:30 [17:53:30.073]
INFO:
prover-client:proving-broker Proving Broker started
17:53:30 [17:53:30.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:936ad2eed4ec06e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:936ad2eed4ec06e3"}
17:53:30 [17:53:30.074]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFavyIs/1 with map size 1073741824
17:53:30 [17:53:30.075]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.199]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:936ad2eed4ec06e3 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:936ad2eed4ec06e3"}
17:53:30 [17:53:30.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8112866560b2458d epochNumber=10
{"provingJobId":"1:BASE_PARITY:8112866560b2458d"}
17:53:30 [17:53:30.201]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testFavyIs/10 with map size 1073741824
17:53:30 [17:53:30.203]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.225]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:30 [17:53:30.226]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:53:30 [17:53:30.349]
INFO:
prover-client:proving-broker Proving Broker started
17:53:30 [17:53:30.350]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:708c040f6f8cfd5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:708c040f6f8cfd5e"}
17:53:30 [17:53:30.351]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWnzGqt/1 with map size 1073741824
17:53:30 [17:53:30.352]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.528]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:708c040f6f8cfd5e timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:708c040f6f8cfd5e"}
17:53:30 [17:53:30.575]
INFO:
prover-client:proving-broker Proving Broker started
17:53:30 [17:53:30.577]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91668262963c08f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:91668262963c08f5"}
17:53:30 [17:53:30.577]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcnb0ZM/1 with map size 1073741824
17:53:30 [17:53:30.579]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.600]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:91668262963c08f5 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:91668262963c08f5"}
17:53:30 [17:53:30.605]
INFO:
prover-client:proving-broker Proving Broker started
17:53:30 [17:53:30.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54c75254fec24489 epochNumber=1
{"provingJobId":"1:BASE_PARITY:54c75254fec24489"}
17:53:30 [17:53:30.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbOFaEz/1 with map size 1073741824
17:53:30 [17:53:30.610]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.630]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:54c75254fec24489 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:54c75254fec24489"}
17:53:30 [17:53:30.631]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:54c75254fec24489 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:54c75254fec24489"}
17:53:30 [17:53:30.641]
INFO:
prover-client:proving-broker Proving Broker started
17:53:30 [17:53:30.642]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92e7352b7c404034 epochNumber=1
{"provingJobId":"1:BASE_PARITY:92e7352b7c404034"}
17:53:30 [17:53:30.643]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPjVzdH/1 with map size 1073741824
17:53:30 [17:53:30.647]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.677]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:92e7352b7c404034 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:92e7352b7c404034"}
17:53:30 [17:53:30.689]
INFO:
prover-client:proving-broker Proving Broker started
17:53:30 [17:53:30.690]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3829c73af8a2d864 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3829c73af8a2d864"}
17:53:30 [17:53:30.692]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIlYZpp/1 with map size 1073741824
17:53:30 [17:53:30.696]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.722]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3829c73af8a2d864 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3829c73af8a2d864"}
17:53:30 [17:53:30.724]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:202a4e427d752803 epochNumber=3
{"provingJobId":"1:BASE_PARITY:202a4e427d752803"}
17:53:30 [17:53:30.725]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testIlYZpp/3 with map size 1073741824
17:53:30 [17:53:30.727]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.766]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:30 [17:53:30.766]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:53:30 [17:53:30.772]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3829c73af8a2d864 err=test error
{"provingJoId":"1:BASE_PARITY:3829c73af8a2d864"}
17:53:30 [17:53:30.779]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testE1Gx82/1 with map size 1073741824
17:53:30 [17:53:30.783]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.810]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testE1Gx82/2 with map size 1073741824
17:53:30 [17:53:30.816]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.835]
INFO:
prover-client:proving-broker Proving Broker started
17:53:30 [17:53:30.838]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8003af689ee42432 settled=false
{"provingJobId":"1:BASE_PARITY:8003af689ee42432","status":"pending"}
17:53:30 [17:53:30.840]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:696ca5aaa2ae773d settled=false
{"provingJobId":"1:BASE_PARITY:696ca5aaa2ae773d","status":"pending"}
17:53:30 [17:53:30.845]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjfudj6/1 with map size 1073741824
17:53:30 [17:53:30.848]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.873]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjfudj6/2 with map size 1073741824
17:53:30 [17:53:30.879]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.915]
INFO:
prover-client:proving-broker Proving Broker started
17:53:30 [17:53:30.918]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:16054641d62e702e settled=true
{"provingJobId":"1:BASE_PARITY:16054641d62e702e","status":"fulfilled"}
17:53:30 [17:53:30.920]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b1155d6a1bd4934c settled=true
{"provingJobId":"2:BASE_PARITY:b1155d6a1bd4934c","status":"fulfilled"}
17:53:30 [17:53:30.927]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNybPld/1 with map size 1073741824
17:53:30 [17:53:30.931]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.959]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNybPld/2 with map size 1073741824
17:53:30 [17:53:30.970]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:30 [17:53:30.990]
INFO:
prover-client:proving-broker Proving Broker started
17:53:30 [17:53:30.993]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d91939708aba7b5e settled=true
{"provingJobId":"1:BASE_PARITY:d91939708aba7b5e","status":"fulfilled"}
17:53:30 [17:53:30.995]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:17e3df5d0c0f19ef settled=false
{"provingJobId":"1:BASE_PARITY:17e3df5d0c0f19ef","status":"pending"}
17:53:31 [17:53:31.002]
INFO:
prover-client:proving-broker Proving Broker started
17:53:31 [17:53:31.003]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1f4e465b6902b99 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1f4e465b6902b99"}
17:53:31 [17:53:31.004]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLiKsyP/1 with map size 1073741824
17:53:31 [17:53:31.006]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.034]
INFO:
prover-client:proving-broker Proving Broker started
17:53:31 [17:53:31.035]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4c2092944adbecf epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4c2092944adbecf"}
17:53:31 [17:53:31.036]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:d4c2092944adbecf: Error: db error: Error: db error
17:53:31 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:d4c2092944adbecf"}
17:53:31 [17:53:31.041]
INFO:
prover-client:proving-broker Proving Broker started
17:53:31 [17:53:31.042]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c56d231e0776001 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c56d231e0776001"}
17:53:31 [17:53:31.044]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9MfqR2/1 with map size 1073741824
17:53:31 [17:53:31.046]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.067]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5c56d231e0776001 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5c56d231e0776001"}
17:53:31 [17:53:31.067]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5c56d231e0776001 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5c56d231e0776001"}
17:53:31 [17:53:31.079]
INFO:
prover-client:proving-broker Proving Broker started
17:53:31 [17:53:31.080]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc04831285a9119c epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc04831285a9119c"}
17:53:31 [17:53:31.081]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9H3bzi/1 with map size 1073741824
17:53:31 [17:53:31.084]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.104]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fc04831285a9119c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fc04831285a9119c"}
17:53:31 [17:53:31.104]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fc04831285a9119c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fc04831285a9119c"}
17:53:31 [17:53:31.105]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:fc04831285a9119c: Error: db error
17:53:31 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:fc04831285a9119c"}
17:53:31 [17:53:31.109]
INFO:
prover-client:proving-broker Proving Broker started
17:53:31 [17:53:31.110]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f18951577a834021 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f18951577a834021"}
17:53:31 [17:53:31.111]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcObz7K/1 with map size 1073741824
17:53:31 [17:53:31.113]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.135]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f18951577a834021 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f18951577a834021"}
17:53:31 [17:53:31.136]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f18951577a834021 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f18951577a834021"}
17:53:31 [17:53:31.150]
INFO:
prover-client:proving-broker Proving Broker started
17:53:31 [17:53:31.151]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77af5bc4d0ab3d76 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77af5bc4d0ab3d76"}
17:53:31 [17:53:31.152]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test18l8LW/1 with map size 1073741824
17:53:31 [17:53:31.158]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.179]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:77af5bc4d0ab3d76 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:77af5bc4d0ab3d76"}
17:53:31 [17:53:31.179]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:77af5bc4d0ab3d76 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:77af5bc4d0ab3d76"}
17:53:31 [17:53:31.180]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:77af5bc4d0ab3d76 jobErr=test error: Error: db error
17:53:31 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:77af5bc4d0ab3d76"}
17:53:31 [17:53:31.187]
INFO:
prover-client:proving-broker Proving Broker started
17:53:31 [17:53:31.187]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:04dc5a35fa1663a7 not found
{"provingJobId":"1:BASE_PARITY:04dc5a35fa1663a7"}
17:53:31 [17:53:31.194]
INFO:
prover-client:proving-broker Proving Broker started
17:53:31 [17:53:31.194]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:eaba1e0a40197452 err=test error
{"provingJoId":"1:BASE_PARITY:eaba1e0a40197452"}
17:53:31 [17:53:31.199]
INFO:
prover-client:proving-broker Proving Broker started
17:53:31 [17:53:31.209]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9755493b7dd385ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:9755493b7dd385ed"}
17:53:31 [17:53:31.210]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAQTRCj/1 with map size 1073741824
17:53:31 [17:53:31.212]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.230]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9755493b7dd385ed type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9755493b7dd385ed"}
17:53:31 [17:53:31.230]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9755493b7dd385ed type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9755493b7dd385ed"}
17:53:31 [17:53:31.246]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:2a449edff21277f9 epochNumber=2
{"provingJobId":"2:BASE_PARITY:2a449edff21277f9"}
17:53:31 [17:53:31.247]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAQTRCj/2 with map size 1073741824
17:53:31 [17:53:31.251]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.269]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:2a449edff21277f9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:2a449edff21277f9"}
17:53:31 [17:53:31.270]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2a449edff21277f9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:2a449edff21277f9"}
17:53:31 [17:53:31.337]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:0ced379ccf453262 epochNumber=3
{"provingJobId":"3:BASE_PARITY:0ced379ccf453262"}
17:53:31 [17:53:31.339]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testAQTRCj/3 with map size 1073741824
17:53:31 [17:53:31.343]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.376]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:31 [17:53:31.377]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:53:31 [17:53:31.427]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:084ca0c63c8620cd epochNumber=4
{"provingJobId":"4:BASE_PARITY:084ca0c63c8620cd"}
17:53:31 [17:53:31.429]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testAQTRCj/4 with map size 1073741824
17:53:31 [17:53:31.432]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.455]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:31 [17:53:31.455]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
17:53:31 [17:53:31.516]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:273af89e4e1f6aca epochNumber=5
{"provingJobId":"5:BASE_PARITY:273af89e4e1f6aca"}
17:53:31 [17:53:31.517]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testAQTRCj/5 with map size 1073741824
17:53:31 [17:53:31.520]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:53:31 [17:53:31.558]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:53:31 [17:53:31.559]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
17:53:31
PASS src/proving_broker/proving_broker.test.ts (
13.104 s)
17:53:31 ProvingBroker
17:53:31 Producer API
17:53:31
✓ refuses stale jobs (37 ms)
17:53:31
✓ enqueues jobs (3 ms)
17:53:31
✓ ignores duplicate jobs (3 ms)
17:53:31
✓ reports correct status when enqueuing repeat jobs (4 ms)
17:53:31
✓ throws an error in case of duplicate job IDs (24 ms)
17:53:31
✓ returns not-found status for non-existing jobs (2 ms)
17:53:31
✓ cancels jobs in queue (3 ms)
17:53:31
✓ cancels jobs in-progress (2 ms)
17:53:31
✓ returns job result if successful (2 ms)
17:53:31
✓ returns job error if failed (2 ms)
17:53:31
✓ correctly returns job status for concurrent writes (2 ms)
17:53:31
✓ refuses stale jobs (37 ms)
17:53:31
✓ enqueues jobs (35 ms)
17:53:31
✓ ignores duplicate jobs (27 ms)
17:53:31
✓ reports correct status when enqueuing repeat jobs (30 ms)
17:53:31
✓ throws an error in case of duplicate job IDs (25 ms)
17:53:31
✓ returns not-found status for non-existing jobs (2 ms)
17:53:31
✓ cancels jobs in queue (29 ms)
17:53:31
✓ cancels jobs in-progress (30 ms)
17:53:31
✓ returns job result if successful (27 ms)
17:53:31
✓ returns job error if failed (28 ms)
17:53:31
✓ correctly returns job status for concurrent writes (38 ms)
17:53:31 Consumer API
17:53:31
✓ returns undefined if no jobs are available (1 ms)
17:53:31
✓ returns jobs in priority order (1 ms)
17:53:31
✓ returns undefined if no jobs are available for the given allowList (2 ms)
17:53:31
✓ returns a job if it is in the allowList (10 ms)
17:53:31
✓ returns the most important job if it is in the allowList (2 ms)
17:53:31
✓ returns any job if filter is empty (3 ms)
17:53:31
✓ returns a new job when reporting job success (2 ms)
17:53:31
✓ returns a new job when reporting permanent error (2 ms)
17:53:31
✓ returns a new job when reporting retry-able error (2 ms)
17:53:31
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
17:53:31
✓ returns a new job if job is already in progress elsewhere (1056 ms)
17:53:31
✓ avoids sending the same job to a new agent after a restart (1005 ms)
17:53:31
✓ avoids sending a completed job to a new agent after a restart (1006 ms)
17:53:31
✓ tracks job result if in progress (9 ms)
17:53:31
✓ tracks job result even if job is in queue (2 ms)
17:53:31
✓ ignores reported job error if unknown job (1 ms)
17:53:31
✓ ignores job result if unknown job (1 ms)
17:53:31
✓ returns undefined if no jobs are available (3 ms)
17:53:31
✓ returns jobs in priority order (50 ms)
17:53:31
✓ returns undefined if no jobs are available for the given allowList (29 ms)
17:53:31
✓ returns a job if it is in the allowList (62 ms)
17:53:31
✓ returns the most important job if it is in the allowList (62 ms)
17:53:31
✓ returns any job if filter is empty (55 ms)
17:53:31
✓ returns a new job when reporting job success (40 ms)
17:53:31
✓ returns a new job when reporting permanent error (41 ms)
17:53:31
✓ returns a new job when reporting retry-able error (49 ms)
17:53:31
✓ returns a new job when reporting progress if current one is cancelled (46 ms)
17:53:31
✓ returns a new job if job is already in progress elsewhere (1118 ms)
17:53:31
✓ avoids sending the same job to a new agent after a restart (1067 ms)
17:53:31
✓ avoids sending a completed job to a new agent after a restart (1063 ms)
17:53:31
✓ tracks job result if in progress (59 ms)
17:53:31
✓ tracks job result even if job is in queue (56 ms)
17:53:31
✓ ignores reported job error if unknown job (3 ms)
17:53:31
✓ ignores job result if unknown job (5 ms)
17:53:31 Timeouts
17:53:31
✓ tracks in progress jobs (2 ms)
17:53:31
✓ re-enqueues jobs that time out (126 ms)
17:53:31
✓ cancel stale jobs that time out (234 ms)
17:53:31
✓ keeps the jobs in progress while it is alive (203 ms)
17:53:31
✓ tracks in progress jobs (24 ms)
17:53:31
✓ re-enqueues jobs that time out (133 ms)
17:53:31
✓ cancel stale jobs that time out (277 ms)
17:53:31
✓ keeps the jobs in progress while it is alive (224 ms)
17:53:31 Retries
17:53:31
✓ retries jobs (2 ms)
17:53:31
✓ retries up to a maximum number of times (3 ms)
17:53:31
✓ passing retry=false does not retry (2 ms)
17:53:31
✓ does not retry if job is stale (28 ms)
17:53:31
✓ retries jobs (30 ms)
17:53:31
✓ retries up to a maximum number of times (36 ms)
17:53:31
✓ passing retry=false does not retry (48 ms)
17:53:31
✓ does not retry if job is stale (87 ms)
17:53:31 Database management
17:53:31
✓ re-enqueues proof requests on start (3 ms)
17:53:31
✓ restores proof results on start (2 ms)
17:53:31
✓ only re-enqueues unfinished jobs (3 ms)
17:53:31
✓ saves job when enqueued (3 ms)
17:53:31
✓ does not retain job if database fails to save (3 ms)
17:53:31
✓ saves job result (4 ms)
17:53:31
✓ saves result even if database fails to save (3 ms)
17:53:31
✓ saves job error (4 ms)
17:53:31
✓ saves job error even if database fails to save (3 ms)
17:53:31
✓ does not save job result if job is unknown (2 ms)
17:53:31
✓ does not save job error if job is unknown (7 ms)
17:53:31
✓ cleans up old jobs periodically (261 ms)
17:53:31
✓ re-enqueues proof requests on start (68 ms)
17:53:31
✓ restores proof results on start (81 ms)
17:53:31
✓ only re-enqueues unfinished jobs (74 ms)
17:53:31
✓ saves job when enqueued (31 ms)
17:53:31
✓ does not retain job if database fails to save (7 ms)
17:53:31
✓ saves job result (38 ms)
17:53:31
✓ saves result even if database fails to save (30 ms)
17:53:31
✓ saves job error (40 ms)
17:53:31
✓ saves job error even if database fails to save (34 ms)
17:53:31
✓ does not save job result if job is unknown (6 ms)
17:53:31
✓ does not save job error if job is unknown (7 ms)
17:53:31
✓ cleans up old jobs periodically (396 ms)
17:53:31
17:53:31
Test Suites: 1 passed, 1 total
17:53:31
Tests: 96 passed, 96 total
17:53:31
Snapshots: 0 total
17:53:31
Time: 13.211 s
17:53:31
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
17:53:31
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?