Parent Log:
http://ci.aztec-labs.com/d674377334c1b750
Command: 0b621446113df0ea yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/a16ffcb44336b117d56ec8733e0b8bbfc787edf7
Env: REF_NAME=gh-readonly-queue/next/pr-15187-4608046e462faf9ff3fbee265ba34dbecb5d0b31 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 23 14:46:51 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15187_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
14:46:56 [14:46:56.320]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.326]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b06d1658104c338b epochNumber=42
{"provingJobId":"1:BASE_PARITY:b06d1658104c338b"}
14:46:56 [14:46:56.329]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:bfcac45d44587c8c epochNumber=1
{"provingJobId":"1:BASE_PARITY:bfcac45d44587c8c"}
14:46:56 [14:46:56.357]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.358]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b4ac5c86ae0705e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b4ac5c86ae0705e9"}
14:46:56 [14:46:56.359]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e4001e386e99246 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e4001e386e99246"}
14:46:56 [14:46:56.361]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c00d3d444441f162 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c00d3d444441f162"}
14:46:56 [14:46:56.363]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c00d3d444441f162 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c00d3d444441f162"}
14:46:56 [14:46:56.365]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f5307aabb7ece48 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f5307aabb7ece48"}
14:46:56 [14:46:56.366]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4f5307aabb7ece48 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4f5307aabb7ece48"}
14:46:56 [14:46:56.366]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4f5307aabb7ece48 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4f5307aabb7ece48"}
14:46:56 [14:46:56.367]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4f5307aabb7ece48 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4f5307aabb7ece48"}
14:46:56 [14:46:56.368]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.368]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed48f24c722c0b40 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed48f24c722c0b40"}
14:46:56 [14:46:56.391]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.392]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.392]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52f25a32d7602be0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:52f25a32d7602be0"}
14:46:56 [14:46:56.393]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:52f25a32d7602be0
{"provingJobId":"1:BASE_PARITY:52f25a32d7602be0"}
14:46:56 [14:46:56.393]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:52f25a32d7602be0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:52f25a32d7602be0"}
14:46:56 [14:46:56.394]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:52f25a32d7602be0 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:52f25a32d7602be0"}
14:46:56 [14:46:56.395]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.395]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98c8d1dcd80b1064 epochNumber=1
{"provingJobId":"1:BASE_PARITY:98c8d1dcd80b1064"}
14:46:56 [14:46:56.396]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:98c8d1dcd80b1064
{"provingJobId":"1:BASE_PARITY:98c8d1dcd80b1064"}
14:46:56 [14:46:56.396]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:98c8d1dcd80b1064 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:98c8d1dcd80b1064"}
14:46:56 [14:46:56.397]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.398]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c3b222bf8167011 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c3b222bf8167011"}
14:46:56 [14:46:56.398]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8c3b222bf8167011 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8c3b222bf8167011"}
14:46:56 [14:46:56.398]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8c3b222bf8167011 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8c3b222bf8167011"}
14:46:56 [14:46:56.399]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.400]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f12f7ff96f3a661 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f12f7ff96f3a661"}
14:46:56 [14:46:56.400]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9f12f7ff96f3a661 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9f12f7ff96f3a661"}
14:46:56 [14:46:56.401]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9f12f7ff96f3a661 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9f12f7ff96f3a661"}
14:46:56 [14:46:56.402]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.402]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08943d4563eff77c epochNumber=0
{"provingJobId":"1:BASE_PARITY:08943d4563eff77c"}
14:46:56 [14:46:56.402]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:08943d4563eff77c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:08943d4563eff77c"}
14:46:56 [14:46:56.403]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba1a38bc910f8333 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ba1a38bc910f8333"}
14:46:56 [14:46:56.403]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:08943d4563eff77c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:08943d4563eff77c"}
14:46:56 [14:46:56.403]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d43edd357c18897 epochNumber=0
{"provingJobId":"1:BASE_PARITY:0d43edd357c18897"}
14:46:56 [14:46:56.410]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.411]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.411]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e19bcc8ae0acbfb4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e19bcc8ae0acbfb4"}
14:46:56 [14:46:56.411]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16484caf88b841d9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:16484caf88b841d9"}
14:46:56 [14:46:56.413]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.413]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4dd2164a4ff2b8a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4dd2164a4ff2b8a"}
14:46:56 [14:46:56.414]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.414]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35bf8e1b6429614e epochNumber=1
{"provingJobId":"1:BASE_PARITY:35bf8e1b6429614e"}
14:46:56 [14:46:56.414]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f9785d419ed1852 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f9785d419ed1852"}
14:46:56 [14:46:56.415]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc183886da36d916 epochNumber=2
{"provingJobId":"1:BASE_PARITY:bc183886da36d916"}
14:46:56 [14:46:56.415]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c33ab4298316a84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c33ab4298316a84"}
14:46:56 [14:46:56.416]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f47c012d8adb1565 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f47c012d8adb1565"}
14:46:56 [14:46:56.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ddd1132b3917b89a epochNumber=1
{"provingJobId":"1:BASE_PARITY:ddd1132b3917b89a"}
14:46:56 [14:46:56.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ead9309d0671528 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2ead9309d0671528"}
14:46:56 [14:46:56.418]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d28f3453621fd2f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d28f3453621fd2f0"}
14:46:56 [14:46:56.419]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.419]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:807307c9bc505441 epochNumber=1
{"provingJobId":"1:BASE_PARITY:807307c9bc505441"}
14:46:56 [14:46:56.420]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c4d1dd92de9e74e epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c4d1dd92de9e74e"}
14:46:56 [14:46:56.420]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35a7f61c19e40589 epochNumber=2
{"provingJobId":"1:BASE_PARITY:35a7f61c19e40589"}
14:46:56 [14:46:56.420]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:696db2c5b4be5047 epochNumber=1
{"provingJobId":"1:BASE_PARITY:696db2c5b4be5047"}
14:46:56 [14:46:56.422]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.422]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd2ee24e3e768bd3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cd2ee24e3e768bd3"}
14:46:56 [14:46:56.422]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89779b395e690eb8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:89779b395e690eb8"}
14:46:56 [14:46:56.423]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cd2ee24e3e768bd3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cd2ee24e3e768bd3"}
14:46:56 [14:46:56.424]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.424]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7560956a11093939 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7560956a11093939"}
14:46:56 [14:46:56.425]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:018b326d45c4ae58 epochNumber=1
{"provingJobId":"1:BASE_PARITY:018b326d45c4ae58"}
14:46:56 [14:46:56.425]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7560956a11093939 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:7560956a11093939"}
14:46:56 [14:46:56.426]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.426]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea6f7bf1f643f69b epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea6f7bf1f643f69b"}
14:46:56 [14:46:56.426]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76817ae93d177eb4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:76817ae93d177eb4"}
14:46:56 [14:46:56.427]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ea6f7bf1f643f69b type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:ea6f7bf1f643f69b"}
14:46:56 [14:46:56.428]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.428]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:61b6e94d35332865 epochNumber=1
{"provingJobId":"1:BASE_PARITY:61b6e94d35332865"}
14:46:56 [14:46:56.428]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:61b6e94d35332865
{"provingJobId":"1:BASE_PARITY:61b6e94d35332865"}
14:46:56 [14:46:56.429]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:61b6e94d35332865 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:61b6e94d35332865"}
14:46:56 [14:46:56.429]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d236877f97e8b252 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d236877f97e8b252"}
14:46:56 [14:46:56.429]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:61b6e94d35332865 has already been completed
{"provingJobId":"1:BASE_PARITY:61b6e94d35332865"}
14:46:56 [14:46:56.430]
INFO:
prover-client:proving-broker Proving Broker started
14:46:56 [14:46:56.431]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06880d53a38245e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:06880d53a38245e8"}
14:46:56 [14:46:56.431]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cbd2b610ac882539 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cbd2b610ac882539"}
14:46:57 [14:46:57.487]
INFO:
prover-client:proving-broker Proving Broker started
14:46:57 [14:46:57.488]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:06880d53a38245e8 settled=false
{"provingJobId":"1:BASE_PARITY:06880d53a38245e8","status":"pending"}
14:46:57 [14:46:57.488]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cbd2b610ac882539 settled=false
{"provingJobId":"1:BASE_PARITY:cbd2b610ac882539","status":"pending"}
14:46:57 [14:46:57.489]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:06880d53a38245e8 type=BASE_PARITY startedAt=1750690016431 older agent has taken job
{"provingJobId":"1:BASE_PARITY:06880d53a38245e8"}
14:46:57 [14:46:57.489]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:06880d53a38245e8 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:06880d53a38245e8"}
14:46:57 [14:46:57.491]
INFO:
prover-client:proving-broker Proving Broker started
14:46:57 [14:46:57.491]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3a560e362952766 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3a560e362952766"}
14:46:57 [14:46:57.492]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee5c773221b7bdf2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ee5c773221b7bdf2"}
14:46:58 [14:46:58.493]
INFO:
prover-client:proving-broker Proving Broker started
14:46:58 [14:46:58.494]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d3a560e362952766 settled=false
{"provingJobId":"1:BASE_PARITY:d3a560e362952766","status":"pending"}
14:46:58 [14:46:58.495]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ee5c773221b7bdf2 settled=false
{"provingJobId":"1:BASE_PARITY:ee5c773221b7bdf2","status":"pending"}
14:46:58 [14:46:58.496]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d3a560e362952766 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:d3a560e362952766"}
14:46:58 [14:46:58.498]
INFO:
prover-client:proving-broker Proving Broker started
14:46:58 [14:46:58.498]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f231d7e594eadbf epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f231d7e594eadbf"}
14:46:58 [14:46:58.499]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06b16e5a3f041ebe epochNumber=2
{"provingJobId":"1:BASE_PARITY:06b16e5a3f041ebe"}
14:46:59 [14:46:59.502]
INFO:
prover-client:proving-broker Proving Broker started
14:46:59 [14:46:59.502]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9f231d7e594eadbf settled=false
{"provingJobId":"1:BASE_PARITY:9f231d7e594eadbf","status":"pending"}
14:46:59 [14:46:59.503]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:06b16e5a3f041ebe settled=false
{"provingJobId":"1:BASE_PARITY:06b16e5a3f041ebe","status":"pending"}
14:46:59 [14:46:59.503]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9f231d7e594eadbf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9f231d7e594eadbf"}
14:46:59 [14:46:59.504]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9f231d7e594eadbf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9f231d7e594eadbf"}
14:46:59 [14:46:59.505]
INFO:
prover-client:proving-broker Proving Broker started
14:46:59 [14:46:59.505]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:339a8a7e803526b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:339a8a7e803526b1"}
14:46:59 [14:46:59.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cb61a0fe14b3824 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0cb61a0fe14b3824"}
14:46:59 [14:46:59.506]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:339a8a7e803526b1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:339a8a7e803526b1"}
14:46:59 [14:46:59.507]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0cb61a0fe14b3824 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0cb61a0fe14b3824"}
14:46:59 [14:46:59.508]
INFO:
prover-client:proving-broker Proving Broker started
14:46:59 [14:46:59.508]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:186c97ee70914699 epochNumber=1
{"provingJobId":"1:BASE_PARITY:186c97ee70914699"}
14:46:59 [14:46:59.509]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:789ab83795121dc4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:789ab83795121dc4"}
14:46:59 [14:46:59.509]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:186c97ee70914699 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:186c97ee70914699"}
14:46:59 [14:46:59.509]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:186c97ee70914699 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:186c97ee70914699"}
14:46:59 [14:46:59.509]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:789ab83795121dc4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:789ab83795121dc4"}
14:46:59 [14:46:59.511]
INFO:
prover-client:proving-broker Proving Broker started
14:46:59 [14:46:59.512]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0c55a2e564a845d9 err=test error
{"provingJoId":"1:BASE_PARITY:0c55a2e564a845d9"}
14:46:59 [14:46:59.512]
INFO:
prover-client:proving-broker Proving Broker started
14:46:59 [14:46:59.513]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:06543b87498cf32c not found
{"provingJobId":"1:BASE_PARITY:06543b87498cf32c"}
14:46:59 [14:46:59.514]
INFO:
prover-client:proving-broker Proving Broker started
14:46:59 [14:46:59.515]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57e543b1ad16cc06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:57e543b1ad16cc06"}
14:46:59 [14:46:59.516]
INFO:
prover-client:proving-broker Proving Broker started
14:46:59 [14:46:59.516]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:489019a6d3c38b17 epochNumber=1
{"provingJobId":"1:BASE_PARITY:489019a6d3c38b17"}
14:46:59 [14:46:59.618]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:489019a6d3c38b17 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:489019a6d3c38b17"}
14:46:59 [14:46:59.625]
INFO:
prover-client:proving-broker Proving Broker started
14:46:59 [14:46:59.626]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80ae183d4a3bad06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:80ae183d4a3bad06"}
14:46:59 [14:46:59.728]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:80ae183d4a3bad06 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:80ae183d4a3bad06"}
14:46:59 [14:46:59.735]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eee8fb542a84eadd epochNumber=10
{"provingJobId":"1:BASE_PARITY:eee8fb542a84eadd"}
14:46:59 [14:46:59.755]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:46:59 [14:46:59.861]
INFO:
prover-client:proving-broker Proving Broker started
14:46:59 [14:46:59.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9c5644fbe7500d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9c5644fbe7500d2"}
14:47:00 [14:47:00.015]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e9c5644fbe7500d2 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e9c5644fbe7500d2"}
14:47:00 [14:47:00.066]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.067]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1cd381885275a499 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1cd381885275a499"}
14:47:00 [14:47:00.067]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1cd381885275a499 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1cd381885275a499"}
14:47:00 [14:47:00.069]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.069]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:241b531b39ae7a67 epochNumber=1
{"provingJobId":"1:BASE_PARITY:241b531b39ae7a67"}
14:47:00 [14:47:00.070]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:241b531b39ae7a67 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:241b531b39ae7a67"}
14:47:00 [14:47:00.071]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:241b531b39ae7a67 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:241b531b39ae7a67"}
14:47:00 [14:47:00.075]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.075]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a53dc3ed2e88d048 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a53dc3ed2e88d048"}
14:47:00 [14:47:00.076]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a53dc3ed2e88d048 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a53dc3ed2e88d048"}
14:47:00 [14:47:00.077]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6fed3dbc035eb2a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6fed3dbc035eb2a0"}
14:47:00 [14:47:00.078]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6fed3dbc035eb2a0 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6fed3dbc035eb2a0"}
14:47:00 [14:47:00.078]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f6499438a3b21b22 epochNumber=3
{"provingJobId":"1:BASE_PARITY:f6499438a3b21b22"}
14:47:00 [14:47:00.103]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:47:00 [14:47:00.104]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6fed3dbc035eb2a0 err=test error
{"provingJoId":"1:BASE_PARITY:6fed3dbc035eb2a0"}
14:47:00 [14:47:00.106]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.106]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:91f17d8c86a629c4 settled=false
{"provingJobId":"1:BASE_PARITY:91f17d8c86a629c4","status":"pending"}
14:47:00 [14:47:00.106]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7f03ac0f32993d3d settled=false
{"provingJobId":"1:BASE_PARITY:7f03ac0f32993d3d","status":"pending"}
14:47:00 [14:47:00.108]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.108]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:90fbc347549ca0e9 settled=true
{"provingJobId":"1:BASE_PARITY:90fbc347549ca0e9","status":"fulfilled"}
14:47:00 [14:47:00.108]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:10cd1117e1447399 settled=true
{"provingJobId":"2:BASE_PARITY:10cd1117e1447399","status":"fulfilled"}
14:47:00 [14:47:00.109]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.109]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f181c2a0ea8e4be4 settled=true
{"provingJobId":"1:BASE_PARITY:f181c2a0ea8e4be4","status":"fulfilled"}
14:47:00 [14:47:00.109]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:88bca6932414f375 settled=false
{"provingJobId":"1:BASE_PARITY:88bca6932414f375","status":"pending"}
14:47:00 [14:47:00.111]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.112]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45b2203f73e62073 epochNumber=1
{"provingJobId":"1:BASE_PARITY:45b2203f73e62073"}
14:47:00 [14:47:00.114]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.115]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f254a310e064b7ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:f254a310e064b7ec"}
14:47:00 [14:47:00.116]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:f254a310e064b7ec: Error: db error: Error: db error
14:47:00 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:f254a310e064b7ec"}
14:47:00 [14:47:00.118]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:feeaefcde91d0837 epochNumber=1
{"provingJobId":"1:BASE_PARITY:feeaefcde91d0837"}
14:47:00 [14:47:00.119]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:feeaefcde91d0837 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:feeaefcde91d0837"}
14:47:00 [14:47:00.119]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:feeaefcde91d0837 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:feeaefcde91d0837"}
14:47:00 [14:47:00.121]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.121]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd9f9692af1ff374 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd9f9692af1ff374"}
14:47:00 [14:47:00.121]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dd9f9692af1ff374 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dd9f9692af1ff374"}
14:47:00 [14:47:00.122]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dd9f9692af1ff374 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dd9f9692af1ff374"}
14:47:00 [14:47:00.122]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:dd9f9692af1ff374: Error: db error
14:47:00 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:dd9f9692af1ff374"}
14:47:00 [14:47:00.123]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f740da505359cab epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f740da505359cab"}
14:47:00 [14:47:00.124]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3f740da505359cab type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3f740da505359cab"}
14:47:00 [14:47:00.124]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3f740da505359cab type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3f740da505359cab"}
14:47:00 [14:47:00.126]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.126]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0e8b62d6438664d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0e8b62d6438664d"}
14:47:00 [14:47:00.126]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b0e8b62d6438664d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b0e8b62d6438664d"}
14:47:00 [14:47:00.127]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b0e8b62d6438664d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b0e8b62d6438664d"}
14:47:00 [14:47:00.127]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:b0e8b62d6438664d jobErr=test error: Error: db error
14:47:00 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:b0e8b62d6438664d"}
14:47:00 [14:47:00.128]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.129]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d5d7397cacb6d2f1 not found
{"provingJobId":"1:BASE_PARITY:d5d7397cacb6d2f1"}
14:47:00 [14:47:00.130]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.131]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6142476e9b8387c7 err=test error
{"provingJoId":"1:BASE_PARITY:6142476e9b8387c7"}
14:47:00 [14:47:00.132]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.142]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b142f8d405faffe epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b142f8d405faffe"}
14:47:00 [14:47:00.143]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4b142f8d405faffe type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4b142f8d405faffe"}
14:47:00 [14:47:00.143]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4b142f8d405faffe type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4b142f8d405faffe"}
14:47:00 [14:47:00.154]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:fb090ae884a0048d epochNumber=2
{"provingJobId":"2:BASE_PARITY:fb090ae884a0048d"}
14:47:00 [14:47:00.155]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:fb090ae884a0048d type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:fb090ae884a0048d"}
14:47:00 [14:47:00.155]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:fb090ae884a0048d type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:fb090ae884a0048d"}
14:47:00 [14:47:00.218]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:4513ce71032888ae epochNumber=3
{"provingJobId":"3:BASE_PARITY:4513ce71032888ae"}
14:47:00 [14:47:00.233]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:47:00 [14:47:00.279]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:0240e4f0dbf94a4e epochNumber=4
{"provingJobId":"4:BASE_PARITY:0240e4f0dbf94a4e"}
14:47:00 [14:47:00.285]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:47:00 [14:47:00.342]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:cd88a5dee41dd1e1 epochNumber=5
{"provingJobId":"5:BASE_PARITY:cd88a5dee41dd1e1"}
14:47:00 [14:47:00.361]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:47:00 [14:47:00.397]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.397]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6bb051199cd21bed epochNumber=42
{"provingJobId":"1:BASE_PARITY:6bb051199cd21bed"}
14:47:00 [14:47:00.398]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testkqWBeR/42 with map size 1073741824
14:47:00 [14:47:00.407]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.436]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:ede5092456bb39f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ede5092456bb39f3"}
14:47:00 [14:47:00.441]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.442]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6aa597976035c503 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6aa597976035c503"}
14:47:00 [14:47:00.443]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0bI0Sa/1 with map size 1073741824
14:47:00 [14:47:00.445]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.465]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b2c088438ad011a epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b2c088438ad011a"}
14:47:00 [14:47:00.474]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.475]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98319d01c6d1ed7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:98319d01c6d1ed7d"}
14:47:00 [14:47:00.476]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlvMlqH/1 with map size 1073741824
14:47:00 [14:47:00.479]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.496]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:98319d01c6d1ed7d epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:98319d01c6d1ed7d"}
14:47:00 [14:47:00.499]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:29579d659778533c epochNumber=1
{"provingJobId":"1:BASE_PARITY:29579d659778533c"}
14:47:00 [14:47:00.500]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWWxx8H/1 with map size 1073741824
14:47:00 [14:47:00.505]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.523]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:29579d659778533c epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:29579d659778533c"}
14:47:00 [14:47:00.524]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:29579d659778533c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:29579d659778533c"}
14:47:00 [14:47:00.529]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:29579d659778533c epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:29579d659778533c"}
14:47:00 [14:47:00.533]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9fe1ed8d5c2a035a epochNumber=1
{"provingJobId":"1:BASE_PARITY:9fe1ed8d5c2a035a"}
14:47:00 [14:47:00.535]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgMZDz3/1 with map size 1073741824
14:47:00 [14:47:00.537]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.558]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.561]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.561]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:286528ec4b4f1a1b epochNumber=1
{"provingJobId":"1:BASE_PARITY:286528ec4b4f1a1b"}
14:47:00 [14:47:00.562]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGkuQzn/1 with map size 1073741824
14:47:00 [14:47:00.564]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.589]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:286528ec4b4f1a1b
{"provingJobId":"1:BASE_PARITY:286528ec4b4f1a1b"}
14:47:00 [14:47:00.589]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:286528ec4b4f1a1b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:286528ec4b4f1a1b"}
14:47:00 [14:47:00.590]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:286528ec4b4f1a1b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:286528ec4b4f1a1b"}
14:47:00 [14:47:00.600]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.600]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2faf2b744c314aae epochNumber=1
{"provingJobId":"1:BASE_PARITY:2faf2b744c314aae"}
14:47:00 [14:47:00.602]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMFUu53/1 with map size 1073741824
14:47:00 [14:47:00.604]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.621]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2faf2b744c314aae
{"provingJobId":"1:BASE_PARITY:2faf2b744c314aae"}
14:47:00 [14:47:00.622]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2faf2b744c314aae type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2faf2b744c314aae"}
14:47:00 [14:47:00.634]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.635]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87906fd581aa14cf epochNumber=1
{"provingJobId":"1:BASE_PARITY:87906fd581aa14cf"}
14:47:00 [14:47:00.636]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPf9QhT/1 with map size 1073741824
14:47:00 [14:47:00.641]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.661]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:87906fd581aa14cf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:87906fd581aa14cf"}
14:47:00 [14:47:00.662]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:87906fd581aa14cf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:87906fd581aa14cf"}
14:47:00 [14:47:00.668]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.669]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:145a5dacedb26375 epochNumber=1
{"provingJobId":"1:BASE_PARITY:145a5dacedb26375"}
14:47:00 [14:47:00.671]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testye1GST/1 with map size 1073741824
14:47:00 [14:47:00.683]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.695]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:145a5dacedb26375 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:145a5dacedb26375"}
14:47:00 [14:47:00.696]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:145a5dacedb26375 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:145a5dacedb26375"}
14:47:00 [14:47:00.702]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.702]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f874dcd02ed326c7 epochNumber=0
{"provingJobId":"1:BASE_PARITY:f874dcd02ed326c7"}
14:47:00 [14:47:00.703]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testcbQYRz/0 with map size 1073741824
14:47:00 [14:47:00.705]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.719]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f874dcd02ed326c7 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f874dcd02ed326c7"}
14:47:00 [14:47:00.720]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7374719589abeb43 epochNumber=0
{"provingJobId":"1:BASE_PARITY:7374719589abeb43"}
14:47:00 [14:47:00.720]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f874dcd02ed326c7 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f874dcd02ed326c7"}
14:47:00 [14:47:00.720]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:016bfe4df3ed1ee5 epochNumber=0
{"provingJobId":"1:BASE_PARITY:016bfe4df3ed1ee5"}
14:47:00 [14:47:00.737]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.739]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4473f5ec32e83d4e epochNumber=2
{"provingJobId":"1:BASE_PARITY:4473f5ec32e83d4e"}
14:47:00 [14:47:00.741]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSwysev/2 with map size 1073741824
14:47:00 [14:47:00.742]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.759]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3f35889b234fbdc epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3f35889b234fbdc"}
14:47:00 [14:47:00.760]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSwysev/1 with map size 1073741824
14:47:00 [14:47:00.761]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.785]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b42cd9da9ce92b66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b42cd9da9ce92b66"}
14:47:00 [14:47:00.787]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWZR4VF/1 with map size 1073741824
14:47:00 [14:47:00.790]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.806]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:187489f931cb7591 epochNumber=1
{"provingJobId":"1:BASE_PARITY:187489f931cb7591"}
14:47:00 [14:47:00.807]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJb0Ilf/1 with map size 1073741824
14:47:00 [14:47:00.809]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.825]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5153939b66d1570a epochNumber=1
{"provingJobId":"1:BASE_PARITY:5153939b66d1570a"}
14:47:00 [14:47:00.831]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d42ddac962ee61e4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d42ddac962ee61e4"}
14:47:00 [14:47:00.832]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJb0Ilf/2 with map size 1073741824
14:47:00 [14:47:00.833]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.850]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b042b754b96299ee epochNumber=1
{"provingJobId":"1:BASE_PARITY:b042b754b96299ee"}
14:47:00 [14:47:00.858]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.859]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa51fd26f21119fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa51fd26f21119fe"}
14:47:00 [14:47:00.859]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTBZgzC/1 with map size 1073741824
14:47:00 [14:47:00.861]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.881]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:173f4abe0628e926 epochNumber=1
{"provingJobId":"1:BASE_PARITY:173f4abe0628e926"}
14:47:00 [14:47:00.888]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd441d7421b8c429 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dd441d7421b8c429"}
14:47:00 [14:47:00.889]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTBZgzC/2 with map size 1073741824
14:47:00 [14:47:00.891]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.907]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7a306dcf3141d6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7a306dcf3141d6c"}
14:47:00 [14:47:00.917]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.918]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55498c8803cccd63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:55498c8803cccd63"}
14:47:00 [14:47:00.918]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsbEQ1j/1 with map size 1073741824
14:47:00 [14:47:00.920]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.938]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5bddaa00a77ff9ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:5bddaa00a77ff9ec"}
14:47:00 [14:47:00.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b50aad00df62bf00 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b50aad00df62bf00"}
14:47:00 [14:47:00.946]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testsbEQ1j/2 with map size 1073741824
14:47:00 [14:47:00.949]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:00 [14:47:00.964]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9bb358d88a6082d epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9bb358d88a6082d"}
14:47:00 [14:47:00.977]
INFO:
prover-client:proving-broker Proving Broker started
14:47:00 [14:47:00.978]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb6515b1f3b591e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb6515b1f3b591e7"}
14:47:00 [14:47:00.979]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTrL5HZ/1 with map size 1073741824
14:47:00 [14:47:00.986]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:01 [14:47:01.003]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:446f0288f09a8621 epochNumber=1
{"provingJobId":"1:BASE_PARITY:446f0288f09a8621"}
14:47:01 [14:47:01.009]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bb6515b1f3b591e7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bb6515b1f3b591e7"}
14:47:01 [14:47:01.021]
INFO:
prover-client:proving-broker Proving Broker started
14:47:01 [14:47:01.022]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93b84d8991868341 epochNumber=1
{"provingJobId":"1:BASE_PARITY:93b84d8991868341"}
14:47:01 [14:47:01.023]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnGqS2i/1 with map size 1073741824
14:47:01 [14:47:01.026]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:01 [14:47:01.044]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21c56b8dd4fddf04 epochNumber=1
{"provingJobId":"1:BASE_PARITY:21c56b8dd4fddf04"}
14:47:01 [14:47:01.050]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:93b84d8991868341 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:93b84d8991868341"}
14:47:01 [14:47:01.059]
INFO:
prover-client:proving-broker Proving Broker started
14:47:01 [14:47:01.060]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:73e0b95d94484560 epochNumber=1
{"provingJobId":"1:BASE_PARITY:73e0b95d94484560"}
14:47:01 [14:47:01.061]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIeUpMR/1 with map size 1073741824
14:47:01 [14:47:01.063]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:01 [14:47:01.081]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5af42a5e644319fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:5af42a5e644319fb"}
14:47:01 [14:47:01.088]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:73e0b95d94484560 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:73e0b95d94484560"}
14:47:01 [14:47:01.094]
INFO:
prover-client:proving-broker Proving Broker started
14:47:01 [14:47:01.095]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7ff1bf6c5e777847 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7ff1bf6c5e777847"}
14:47:01 [14:47:01.095]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpDzstZ/1 with map size 1073741824
14:47:01 [14:47:01.097]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:01 [14:47:01.118]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7ff1bf6c5e777847
{"provingJobId":"1:BASE_PARITY:7ff1bf6c5e777847"}
14:47:01 [14:47:01.119]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7ff1bf6c5e777847 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7ff1bf6c5e777847"}
14:47:01 [14:47:01.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa67d75b9eb8b626 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa67d75b9eb8b626"}
14:47:01 [14:47:01.133]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7ff1bf6c5e777847 has already been completed
{"provingJobId":"1:BASE_PARITY:7ff1bf6c5e777847"}
14:47:01 [14:47:01.136]
INFO:
prover-client:proving-broker Proving Broker started
14:47:01 [14:47:01.137]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41bb68eb6f16b4a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:41bb68eb6f16b4a6"}
14:47:01 [14:47:01.137]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdM8jP3/1 with map size 1073741824
14:47:01 [14:47:01.141]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:01 [14:47:01.160]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2071102ae12ff51c epochNumber=2
{"provingJobId":"1:BASE_PARITY:2071102ae12ff51c"}
14:47:01 [14:47:01.162]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testdM8jP3/2 with map size 1073741824
14:47:01 [14:47:01.164]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:02 [14:47:02.245]
INFO:
prover-client:proving-broker Proving Broker started
14:47:02 [14:47:02.253]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:41bb68eb6f16b4a6 settled=false
{"provingJobId":"1:BASE_PARITY:41bb68eb6f16b4a6","status":"pending"}
14:47:02 [14:47:02.255]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2071102ae12ff51c settled=false
{"provingJobId":"1:BASE_PARITY:2071102ae12ff51c","status":"pending"}
14:47:02 [14:47:02.256]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:41bb68eb6f16b4a6 type=BASE_PARITY startedAt=1750690021192 older agent has taken job
{"provingJobId":"1:BASE_PARITY:41bb68eb6f16b4a6"}
14:47:02 [14:47:02.256]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:41bb68eb6f16b4a6 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:41bb68eb6f16b4a6"}
14:47:02 [14:47:02.260]
INFO:
prover-client:proving-broker Proving Broker started
14:47:02 [14:47:02.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc572c0a4b2b46fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc572c0a4b2b46fd"}
14:47:02 [14:47:02.262]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1LflqK/1 with map size 1073741824
14:47:02 [14:47:02.264]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:02 [14:47:02.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a8082a9d55e5d719 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a8082a9d55e5d719"}
14:47:02 [14:47:02.287]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test1LflqK/2 with map size 1073741824
14:47:02 [14:47:02.289]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:03 [14:47:03.307]
INFO:
prover-client:proving-broker Proving Broker started
14:47:03 [14:47:03.310]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cc572c0a4b2b46fd settled=false
{"provingJobId":"1:BASE_PARITY:cc572c0a4b2b46fd","status":"pending"}
14:47:03 [14:47:03.311]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a8082a9d55e5d719 settled=false
{"provingJobId":"1:BASE_PARITY:a8082a9d55e5d719","status":"pending"}
14:47:03 [14:47:03.312]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cc572c0a4b2b46fd type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:cc572c0a4b2b46fd"}
14:47:03 [14:47:03.316]
INFO:
prover-client:proving-broker Proving Broker started
14:47:03 [14:47:03.316]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c3e6222f2c11aee4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c3e6222f2c11aee4"}
14:47:03 [14:47:03.317]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqjNOyt/1 with map size 1073741824
14:47:03 [14:47:03.318]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:03 [14:47:03.336]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97f7ef84bc1a3e6e epochNumber=2
{"provingJobId":"1:BASE_PARITY:97f7ef84bc1a3e6e"}
14:47:03 [14:47:03.337]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqjNOyt/2 with map size 1073741824
14:47:03 [14:47:03.338]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:04 [14:47:04.356]
INFO:
prover-client:proving-broker Proving Broker started
14:47:04 [14:47:04.360]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c3e6222f2c11aee4 settled=false
{"provingJobId":"1:BASE_PARITY:c3e6222f2c11aee4","status":"pending"}
14:47:04 [14:47:04.362]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:97f7ef84bc1a3e6e settled=false
{"provingJobId":"1:BASE_PARITY:97f7ef84bc1a3e6e","status":"pending"}
14:47:04 [14:47:04.363]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c3e6222f2c11aee4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c3e6222f2c11aee4"}
14:47:04 [14:47:04.364]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c3e6222f2c11aee4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c3e6222f2c11aee4"}
14:47:04 [14:47:04.377]
INFO:
prover-client:proving-broker Proving Broker started
14:47:04 [14:47:04.378]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45933db24e1cb468 epochNumber=1
{"provingJobId":"1:BASE_PARITY:45933db24e1cb468"}
14:47:04 [14:47:04.379]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLMkPcS/1 with map size 1073741824
14:47:04 [14:47:04.381]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:04 [14:47:04.404]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a87ac51987e6bcc3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a87ac51987e6bcc3"}
14:47:04 [14:47:04.405]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testLMkPcS/2 with map size 1073741824
14:47:04 [14:47:04.407]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:04 [14:47:04.428]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:45933db24e1cb468 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:45933db24e1cb468"}
14:47:04 [14:47:04.436]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a87ac51987e6bcc3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a87ac51987e6bcc3"}
14:47:04 [14:47:04.446]
INFO:
prover-client:proving-broker Proving Broker started
14:47:04 [14:47:04.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7673c60f55d7dcbe epochNumber=1
{"provingJobId":"1:BASE_PARITY:7673c60f55d7dcbe"}
14:47:04 [14:47:04.448]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7lMF1T/1 with map size 1073741824
14:47:04 [14:47:04.450]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:04 [14:47:04.466]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f179d3c8649fd5ba epochNumber=2
{"provingJobId":"1:BASE_PARITY:f179d3c8649fd5ba"}
14:47:04 [14:47:04.467]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7lMF1T/2 with map size 1073741824
14:47:04 [14:47:04.470]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:04 [14:47:04.488]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7673c60f55d7dcbe type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7673c60f55d7dcbe"}
14:47:04 [14:47:04.489]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7673c60f55d7dcbe type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7673c60f55d7dcbe"}
14:47:04 [14:47:04.495]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f179d3c8649fd5ba type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f179d3c8649fd5ba"}
14:47:04 [14:47:04.504]
INFO:
prover-client:proving-broker Proving Broker started
14:47:04 [14:47:04.505]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a3fc9e876db94abf err=test error
{"provingJoId":"1:BASE_PARITY:a3fc9e876db94abf"}
14:47:04 [14:47:04.508]
INFO:
prover-client:proving-broker Proving Broker started
14:47:04 [14:47:04.508]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2587fdd1fb6dc139 not found
{"provingJobId":"1:BASE_PARITY:2587fdd1fb6dc139"}
14:47:04 [14:47:04.511]
INFO:
prover-client:proving-broker Proving Broker started
14:47:04 [14:47:04.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ab45ee267a5e046 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ab45ee267a5e046"}
14:47:04 [14:47:04.513]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJOcZzz/1 with map size 1073741824
14:47:04 [14:47:04.514]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:04 [14:47:04.536]
INFO:
prover-client:proving-broker Proving Broker started
14:47:04 [14:47:04.536]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f63f240224e9e4d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f63f240224e9e4d2"}
14:47:04 [14:47:04.537]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8SK2Gr/1 with map size 1073741824
14:47:04 [14:47:04.539]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:04 [14:47:04.663]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f63f240224e9e4d2 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f63f240224e9e4d2"}
14:47:04 [14:47:04.672]
INFO:
prover-client:proving-broker Proving Broker started
14:47:04 [14:47:04.673]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:524fdbe3068acb3d epochNumber=1
{"provingJobId":"1:BASE_PARITY:524fdbe3068acb3d"}
14:47:04 [14:47:04.676]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiqVVH0/1 with map size 1073741824
14:47:04 [14:47:04.679]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:04 [14:47:04.799]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:524fdbe3068acb3d timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:524fdbe3068acb3d"}
14:47:04 [14:47:04.803]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76e9c091d35c3d01 epochNumber=10
{"provingJobId":"1:BASE_PARITY:76e9c091d35c3d01"}
14:47:04 [14:47:04.804]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testiqVVH0/10 with map size 1073741824
14:47:04 [14:47:04.808]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:04 [14:47:04.852]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:47:04 [14:47:04.852]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:47:04 [14:47:04.962]
INFO:
prover-client:proving-broker Proving Broker started
14:47:04 [14:47:04.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a229cde67fbc8eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a229cde67fbc8eb"}
14:47:04 [14:47:04.964]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLJoCIC/1 with map size 1073741824
14:47:04 [14:47:04.967]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.140]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5a229cde67fbc8eb timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5a229cde67fbc8eb"}
14:47:05 [14:47:05.194]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.195]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:570e65bbdcbc7a49 epochNumber=1
{"provingJobId":"1:BASE_PARITY:570e65bbdcbc7a49"}
14:47:05 [14:47:05.196]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testa8FcuZ/1 with map size 1073741824
14:47:05 [14:47:05.198]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.218]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:570e65bbdcbc7a49 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:570e65bbdcbc7a49"}
14:47:05 [14:47:05.226]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.226]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d71e4ebbffd8c9cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:d71e4ebbffd8c9cd"}
14:47:05 [14:47:05.227]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtryDGC/1 with map size 1073741824
14:47:05 [14:47:05.229]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.247]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d71e4ebbffd8c9cd type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d71e4ebbffd8c9cd"}
14:47:05 [14:47:05.248]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d71e4ebbffd8c9cd type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:d71e4ebbffd8c9cd"}
14:47:05 [14:47:05.257]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.258]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:972dda198407d2dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:972dda198407d2dc"}
14:47:05 [14:47:05.258]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJRm3Ju/1 with map size 1073741824
14:47:05 [14:47:05.260]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.276]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:972dda198407d2dc type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:972dda198407d2dc"}
14:47:05 [14:47:05.286]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c6ca8d36e020fe8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c6ca8d36e020fe8"}
14:47:05 [14:47:05.288]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgkItVO/1 with map size 1073741824
14:47:05 [14:47:05.290]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.306]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6c6ca8d36e020fe8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6c6ca8d36e020fe8"}
14:47:05 [14:47:05.307]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0af63779ab2c1f76 epochNumber=3
{"provingJobId":"1:BASE_PARITY:0af63779ab2c1f76"}
14:47:05 [14:47:05.307]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testgkItVO/3 with map size 1073741824
14:47:05 [14:47:05.309]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.336]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:47:05 [14:47:05.336]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:47:05 [14:47:05.350]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6c6ca8d36e020fe8 err=test error
{"provingJoId":"1:BASE_PARITY:6c6ca8d36e020fe8"}
14:47:05 [14:47:05.357]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHcYRy4/1 with map size 1073741824
14:47:05 [14:47:05.365]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.391]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testHcYRy4/2 with map size 1073741824
14:47:05 [14:47:05.394]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.411]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.413]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4dd4f22532ff0a6d settled=false
{"provingJobId":"1:BASE_PARITY:4dd4f22532ff0a6d","status":"pending"}
14:47:05 [14:47:05.414]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:98f2ac4fa278db2a settled=false
{"provingJobId":"1:BASE_PARITY:98f2ac4fa278db2a","status":"pending"}
14:47:05 [14:47:05.420]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test84LOSG/1 with map size 1073741824
14:47:05 [14:47:05.422]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.442]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test84LOSG/2 with map size 1073741824
14:47:05 [14:47:05.445]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.476]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.483]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3b8a278196d0b169 settled=true
{"provingJobId":"1:BASE_PARITY:3b8a278196d0b169","status":"fulfilled"}
14:47:05 [14:47:05.486]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:50c7a7b585a4e14c settled=true
{"provingJobId":"2:BASE_PARITY:50c7a7b585a4e14c","status":"fulfilled"}
14:47:05 [14:47:05.492]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpoSHON/1 with map size 1073741824
14:47:05 [14:47:05.494]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.519]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpoSHON/2 with map size 1073741824
14:47:05 [14:47:05.522]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.538]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.540]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c916982823b0752f settled=true
{"provingJobId":"1:BASE_PARITY:c916982823b0752f","status":"fulfilled"}
14:47:05 [14:47:05.542]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6590712af1ed3108 settled=false
{"provingJobId":"1:BASE_PARITY:6590712af1ed3108","status":"pending"}
14:47:05 [14:47:05.548]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.549]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa6b535262cccfaa epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa6b535262cccfaa"}
14:47:05 [14:47:05.549]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testm8rWQL/1 with map size 1073741824
14:47:05 [14:47:05.551]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.575]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.576]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab97823e0d884115 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab97823e0d884115"}
14:47:05 [14:47:05.577]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:ab97823e0d884115: Error: db error: Error: db error
14:47:05 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:ab97823e0d884115"}
14:47:05 [14:47:05.583]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.584]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df1ff45ac97003f7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df1ff45ac97003f7"}
14:47:05 [14:47:05.586]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIJSxQH/1 with map size 1073741824
14:47:05 [14:47:05.589]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.609]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:df1ff45ac97003f7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:df1ff45ac97003f7"}
14:47:05 [14:47:05.610]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:df1ff45ac97003f7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:df1ff45ac97003f7"}
14:47:05 [14:47:05.623]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.624]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:623319bb3327d867 epochNumber=1
{"provingJobId":"1:BASE_PARITY:623319bb3327d867"}
14:47:05 [14:47:05.626]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4eelWS/1 with map size 1073741824
14:47:05 [14:47:05.629]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.645]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:623319bb3327d867 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:623319bb3327d867"}
14:47:05 [14:47:05.645]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:623319bb3327d867 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:623319bb3327d867"}
14:47:05 [14:47:05.646]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:623319bb3327d867: Error: db error
14:47:05 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:623319bb3327d867"}
14:47:05 [14:47:05.652]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.652]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a41471bf822a150 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a41471bf822a150"}
14:47:05 [14:47:05.653]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwL4bwt/1 with map size 1073741824
14:47:05 [14:47:05.657]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.682]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7a41471bf822a150 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7a41471bf822a150"}
14:47:05 [14:47:05.684]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7a41471bf822a150 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7a41471bf822a150"}
14:47:05 [14:47:05.695]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.696]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:796303974b3437d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:796303974b3437d0"}
14:47:05 [14:47:05.697]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testukdU34/1 with map size 1073741824
14:47:05 [14:47:05.700]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.720]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:796303974b3437d0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:796303974b3437d0"}
14:47:05 [14:47:05.721]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:796303974b3437d0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:796303974b3437d0"}
14:47:05 [14:47:05.722]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:796303974b3437d0 jobErr=test error: Error: db error
14:47:05 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:796303974b3437d0"}
14:47:05 [14:47:05.728]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.728]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b854b1a460e13a7b not found
{"provingJobId":"1:BASE_PARITY:b854b1a460e13a7b"}
14:47:05 [14:47:05.732]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.733]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f4dfd57fcfe08a28 err=test error
{"provingJoId":"1:BASE_PARITY:f4dfd57fcfe08a28"}
14:47:05 [14:47:05.737]
INFO:
prover-client:proving-broker Proving Broker started
14:47:05 [14:47:05.748]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a245ca7d2c9f6f67 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a245ca7d2c9f6f67"}
14:47:05 [14:47:05.749]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testW45Kkk/1 with map size 1073741824
14:47:05 [14:47:05.752]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.771]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a245ca7d2c9f6f67 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a245ca7d2c9f6f67"}
14:47:05 [14:47:05.771]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a245ca7d2c9f6f67 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a245ca7d2c9f6f67"}
14:47:05 [14:47:05.791]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:3b62620c10af5671 epochNumber=2
{"provingJobId":"2:BASE_PARITY:3b62620c10af5671"}
14:47:05 [14:47:05.792]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testW45Kkk/2 with map size 1073741824
14:47:05 [14:47:05.795]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.813]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:3b62620c10af5671 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:3b62620c10af5671"}
14:47:05 [14:47:05.814]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:3b62620c10af5671 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:3b62620c10af5671"}
14:47:05 [14:47:05.881]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:5882c4adf23d2a6e epochNumber=3
{"provingJobId":"3:BASE_PARITY:5882c4adf23d2a6e"}
14:47:05 [14:47:05.883]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testW45Kkk/3 with map size 1073741824
14:47:05 [14:47:05.886]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:05 [14:47:05.918]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:47:05 [14:47:05.919]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:47:05 [14:47:05.971]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:bb6f4261e4a2b78b epochNumber=4
{"provingJobId":"4:BASE_PARITY:bb6f4261e4a2b78b"}
14:47:05 [14:47:05.973]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testW45Kkk/4 with map size 1073741824
14:47:05 [14:47:05.979]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:06 [14:47:06.023]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:47:06 [14:47:06.024]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
14:47:06 [14:47:06.063]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:5298784537c463a5 epochNumber=5
{"provingJobId":"5:BASE_PARITY:5298784537c463a5"}
14:47:06 [14:47:06.064]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testW45Kkk/5 with map size 1073741824
14:47:06 [14:47:06.079]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:47:06 [14:47:06.129]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:47:06 [14:47:06.130]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
14:47:06
PASS src/proving_broker/proving_broker.test.ts (
13.126 s)
14:47:06 ProvingBroker
14:47:06 Producer API
14:47:06
✓ refuses stale jobs (40 ms)
14:47:06
✓ enqueues jobs (3 ms)
14:47:06
✓ ignores duplicate jobs (3 ms)
14:47:06
✓ reports correct status when enqueuing repeat jobs (3 ms)
14:47:06
✓ throws an error in case of duplicate job IDs (22 ms)
14:47:06
✓ returns not-found status for non-existing jobs (1 ms)
14:47:06
✓ cancels jobs in queue (2 ms)
14:47:06
✓ cancels jobs in-progress (2 ms)
14:47:06
✓ returns job result if successful (3 ms)
14:47:06
✓ returns job error if failed (2 ms)
14:47:06
✓ correctly returns job status for concurrent writes (3 ms)
14:47:06
✓ refuses stale jobs (45 ms)
14:47:06
✓ enqueues jobs (33 ms)
14:47:06
✓ ignores duplicate jobs (24 ms)
14:47:06
✓ reports correct status when enqueuing repeat jobs (32 ms)
14:47:06
✓ throws an error in case of duplicate job IDs (25 ms)
14:47:06
✓ returns not-found status for non-existing jobs (3 ms)
14:47:06
✓ cancels jobs in queue (38 ms)
14:47:06
✓ cancels jobs in-progress (32 ms)
14:47:06
✓ returns job result if successful (36 ms)
14:47:06
✓ returns job error if failed (33 ms)
14:47:06
✓ correctly returns job status for concurrent writes (36 ms)
14:47:06 Consumer API
14:47:06
✓ returns undefined if no jobs are available (6 ms)
14:47:06
✓ returns jobs in priority order (1 ms)
14:47:06
✓ returns undefined if no jobs are available for the given allowList (1 ms)
14:47:06
✓ returns a job if it is in the allowList (3 ms)
14:47:06
✓ returns the most important job if it is in the allowList (2 ms)
14:47:06
✓ returns any job if filter is empty (2 ms)
14:47:06
✓ returns a new job when reporting job success (2 ms)
14:47:06
✓ returns a new job when reporting permanent error (2 ms)
14:47:06
✓ returns a new job when reporting retry-able error (2 ms)
14:47:06
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
14:47:06
✓ returns a new job if job is already in progress elsewhere (1060 ms)
14:47:06
✓ avoids sending the same job to a new agent after a restart (1007 ms)
14:47:06
✓ avoids sending a completed job to a new agent after a restart (1007 ms)
14:47:06
✓ tracks job result if in progress (2 ms)
14:47:06
✓ tracks job result even if job is in queue (2 ms)
14:47:06
✓ ignores reported job error if unknown job (1 ms)
14:47:06
✓ ignores job result if unknown job (1 ms)
14:47:06
✓ returns undefined if no jobs are available (2 ms)
14:47:06
✓ returns jobs in priority order (46 ms)
14:47:06
✓ returns undefined if no jobs are available for the given allowList (21 ms)
14:47:06
✓ returns a job if it is in the allowList (52 ms)
14:47:06
✓ returns the most important job if it is in the allowList (57 ms)
14:47:06
✓ returns any job if filter is empty (58 ms)
14:47:06
✓ returns a new job when reporting job success (42 ms)
14:47:06
✓ returns a new job when reporting permanent error (41 ms)
14:47:06
✓ returns a new job when reporting retry-able error (32 ms)
14:47:06
✓ returns a new job when reporting progress if current one is cancelled (43 ms)
14:47:06
✓ returns a new job if job is already in progress elsewhere (1123 ms)
14:47:06
✓ avoids sending the same job to a new agent after a restart (1056 ms)
14:47:06
✓ avoids sending a completed job to a new agent after a restart (1061 ms)
14:47:06
✓ tracks job result if in progress (68 ms)
14:47:06
✓ tracks job result even if job is in queue (59 ms)
14:47:06
✓ ignores reported job error if unknown job (3 ms)
14:47:06
✓ ignores job result if unknown job (3 ms)
14:47:06 Timeouts
14:47:06
✓ tracks in progress jobs (1 ms)
14:47:06
✓ re-enqueues jobs that time out (110 ms)
14:47:06
✓ cancel stale jobs that time out (236 ms)
14:47:06
✓ keeps the jobs in progress while it is alive (204 ms)
14:47:06
✓ tracks in progress jobs (24 ms)
14:47:06
✓ re-enqueues jobs that time out (134 ms)
14:47:06
✓ cancel stale jobs that time out (292 ms)
14:47:06
✓ keeps the jobs in progress while it is alive (230 ms)
14:47:06 Retries
14:47:06
✓ retries jobs (3 ms)
14:47:06
✓ retries up to a maximum number of times (5 ms)
14:47:06
✓ passing retry=false does not retry (1 ms)
14:47:06
✓ does not retry if job is stale (28 ms)
14:47:06
✓ retries jobs (27 ms)
14:47:06
✓ retries up to a maximum number of times (35 ms)
14:47:06
✓ passing retry=false does not retry (28 ms)
14:47:06
✓ does not retry if job is stale (68 ms)
14:47:06 Database management
14:47:06
✓ re-enqueues proof requests on start (2 ms)
14:47:06
✓ restores proof results on start (2 ms)
14:47:06
✓ only re-enqueues unfinished jobs (1 ms)
14:47:06
✓ saves job when enqueued (2 ms)
14:47:06
✓ does not retain job if database fails to save (4 ms)
14:47:06
✓ saves job result (2 ms)
14:47:06
✓ saves result even if database fails to save (3 ms)
14:47:06
✓ saves job error (2 ms)
14:47:06
✓ saves job error even if database fails to save (3 ms)
14:47:06
✓ does not save job result if job is unknown (2 ms)
14:47:06
✓ does not save job error if job is unknown (1 ms)
14:47:06
✓ cleans up old jobs periodically (263 ms)
14:47:06
✓ re-enqueues proof requests on start (64 ms)
14:47:06
✓ restores proof results on start (71 ms)
14:47:06
✓ only re-enqueues unfinished jobs (57 ms)
14:47:06
✓ saves job when enqueued (26 ms)
14:47:06
✓ does not retain job if database fails to save (5 ms)
14:47:06
✓ saves job result (39 ms)
14:47:06
✓ saves result even if database fails to save (30 ms)
14:47:06
✓ saves job error (44 ms)
14:47:06
✓ saves job error even if database fails to save (32 ms)
14:47:06
✓ does not save job result if job is unknown (4 ms)
14:47:06
✓ does not save job error if job is unknown (4 ms)
14:47:06
✓ cleans up old jobs periodically (426 ms)
14:47:06
14:47:06
Test Suites: 1 passed, 1 total
14:47:06
Tests: 96 passed, 96 total
14:47:06
Snapshots: 0 total
14:47:06
Time: 13.231 s
14:47:06
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
14:47:06
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?