Parent Log:
http://ci.aztec-labs.com/a079291c5cb4bf29
Command: 9e580e2a65a38075 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/184d5f7d956dd7251670a2dd6728f313cfdf3caf
Env: REF_NAME=gh-readonly-queue/next/pr-14822-e56baa7f24bac54baf9e2f22f6f33ae6fa8b8c0f CURRENT_VERSION=0.87.6 CI_FULL=0
Date: Fri Jun 13 21:21:30 UTC 2025
System: ARCH=arm64 CPUS=64 MEM=247Gi HOSTNAME=pr-14822_arm64_a1-fast
Resources: CPU_LIST=0-63 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
21:21:32 [21:21:32.469]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.474]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9180d79cc25bcbba epochNumber=42
{"provingJobId":"1:BASE_PARITY:9180d79cc25bcbba"}
21:21:32 [21:21:32.477]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:7115e21fd66870b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7115e21fd66870b1"}
21:21:32 [21:21:32.500]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.501]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b847e37a4e47fa97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b847e37a4e47fa97"}
21:21:32 [21:21:32.502]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3730ec7c0d06d0c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3730ec7c0d06d0c"}
21:21:32 [21:21:32.504]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.504]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eaf3717b058722b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eaf3717b058722b4"}
21:21:32 [21:21:32.505]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:eaf3717b058722b4 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:eaf3717b058722b4"}
21:21:32 [21:21:32.508]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.508]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:defb1a817d898fa9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:defb1a817d898fa9"}
21:21:32 [21:21:32.509]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:defb1a817d898fa9 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:defb1a817d898fa9"}
21:21:32 [21:21:32.510]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:defb1a817d898fa9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:defb1a817d898fa9"}
21:21:32 [21:21:32.511]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:defb1a817d898fa9 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:defb1a817d898fa9"}
21:21:32 [21:21:32.512]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85bfbcca2eafbe24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:85bfbcca2eafbe24"}
21:21:32 [21:21:32.530]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.536]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47cc07f4220cb807 epochNumber=1
{"provingJobId":"1:BASE_PARITY:47cc07f4220cb807"}
21:21:32 [21:21:32.537]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:47cc07f4220cb807
{"provingJobId":"1:BASE_PARITY:47cc07f4220cb807"}
21:21:32 [21:21:32.538]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:47cc07f4220cb807 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:47cc07f4220cb807"}
21:21:32 [21:21:32.538]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:47cc07f4220cb807 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:47cc07f4220cb807"}
21:21:32 [21:21:32.539]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.540]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b78f1cf5e2b981ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:b78f1cf5e2b981ff"}
21:21:32 [21:21:32.540]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b78f1cf5e2b981ff
{"provingJobId":"1:BASE_PARITY:b78f1cf5e2b981ff"}
21:21:32 [21:21:32.540]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b78f1cf5e2b981ff type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b78f1cf5e2b981ff"}
21:21:32 [21:21:32.542]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.543]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:850f5e7a90b03f5c epochNumber=1
{"provingJobId":"1:BASE_PARITY:850f5e7a90b03f5c"}
21:21:32 [21:21:32.543]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:850f5e7a90b03f5c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:850f5e7a90b03f5c"}
21:21:32 [21:21:32.543]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:850f5e7a90b03f5c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:850f5e7a90b03f5c"}
21:21:32 [21:21:32.545]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.545]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee8e346020c49e69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee8e346020c49e69"}
21:21:32 [21:21:32.545]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ee8e346020c49e69 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ee8e346020c49e69"}
21:21:32 [21:21:32.546]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ee8e346020c49e69 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ee8e346020c49e69"}
21:21:32 [21:21:32.547]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.547]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a092ea3bcd588c1b epochNumber=0
{"provingJobId":"1:BASE_PARITY:a092ea3bcd588c1b"}
21:21:32 [21:21:32.547]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a092ea3bcd588c1b epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:a092ea3bcd588c1b"}
21:21:32 [21:21:32.548]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ad0572fd5def4cf8 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ad0572fd5def4cf8"}
21:21:32 [21:21:32.548]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a092ea3bcd588c1b epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:a092ea3bcd588c1b"}
21:21:32 [21:21:32.548]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5c3f73b466dc54a epochNumber=0
{"provingJobId":"1:BASE_PARITY:e5c3f73b466dc54a"}
21:21:32 [21:21:32.549]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.550]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.550]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c6e38cc76349736 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2c6e38cc76349736"}
21:21:32 [21:21:32.551]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a59e01af5683b9bc epochNumber=1
{"provingJobId":"1:BASE_PARITY:a59e01af5683b9bc"}
21:21:32 [21:21:32.552]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.552]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a8a1ba1e30049331 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a8a1ba1e30049331"}
21:21:32 [21:21:32.553]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.553]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:407ec2a8af99d833 epochNumber=1
{"provingJobId":"1:BASE_PARITY:407ec2a8af99d833"}
21:21:32 [21:21:32.554]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8ce9f6b1a604283 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e8ce9f6b1a604283"}
21:21:32 [21:21:32.554]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b88d11be1c265eb epochNumber=2
{"provingJobId":"1:BASE_PARITY:0b88d11be1c265eb"}
21:21:32 [21:21:32.554]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17093bfc728252b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17093bfc728252b1"}
21:21:32 [21:21:32.555]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.555]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd9c67e561ce35e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd9c67e561ce35e8"}
21:21:32 [21:21:32.556]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79a126eee89626af epochNumber=1
{"provingJobId":"1:BASE_PARITY:79a126eee89626af"}
21:21:32 [21:21:32.556]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca84cb85976ae381 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ca84cb85976ae381"}
21:21:32 [21:21:32.556]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72ae911f0dbd8f14 epochNumber=1
{"provingJobId":"1:BASE_PARITY:72ae911f0dbd8f14"}
21:21:32 [21:21:32.557]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.557]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:643beae803c23425 epochNumber=1
{"provingJobId":"1:BASE_PARITY:643beae803c23425"}
21:21:32 [21:21:32.558]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:806adffd6e468400 epochNumber=1
{"provingJobId":"1:BASE_PARITY:806adffd6e468400"}
21:21:32 [21:21:32.558]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8abe01a840c34d74 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8abe01a840c34d74"}
21:21:32 [21:21:32.558]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14b871ca183fce30 epochNumber=1
{"provingJobId":"1:BASE_PARITY:14b871ca183fce30"}
21:21:32 [21:21:32.559]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.559]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e4fc718a649f915 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e4fc718a649f915"}
21:21:32 [21:21:32.560]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eaff3dc8f287133c epochNumber=1
{"provingJobId":"1:BASE_PARITY:eaff3dc8f287133c"}
21:21:32 [21:21:32.560]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1e4fc718a649f915 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1e4fc718a649f915"}
21:21:32 [21:21:32.561]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.561]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77057059a3a3f3d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77057059a3a3f3d3"}
21:21:32 [21:21:32.562]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6cb539cec7f82e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6cb539cec7f82e3"}
21:21:32 [21:21:32.562]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:77057059a3a3f3d3 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:77057059a3a3f3d3"}
21:21:32 [21:21:32.563]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.563]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6eba8fff99fa29e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6eba8fff99fa29e6"}
21:21:32 [21:21:32.564]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba787ecb8d4774b7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba787ecb8d4774b7"}
21:21:32 [21:21:32.564]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6eba8fff99fa29e6 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:6eba8fff99fa29e6"}
21:21:32 [21:21:32.565]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.565]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6551aea343f6997 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6551aea343f6997"}
21:21:32 [21:21:32.565]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c6551aea343f6997
{"provingJobId":"1:BASE_PARITY:c6551aea343f6997"}
21:21:32 [21:21:32.566]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c6551aea343f6997 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c6551aea343f6997"}
21:21:32 [21:21:32.566]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:30d01e08de873ffc epochNumber=1
{"provingJobId":"1:BASE_PARITY:30d01e08de873ffc"}
21:21:32 [21:21:32.566]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c6551aea343f6997 has already been completed
{"provingJobId":"1:BASE_PARITY:c6551aea343f6997"}
21:21:32 [21:21:32.567]
INFO:
prover-client:proving-broker Proving Broker started
21:21:32 [21:21:32.568]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c05e4a05a9a6661a epochNumber=1
{"provingJobId":"1:BASE_PARITY:c05e4a05a9a6661a"}
21:21:32 [21:21:32.568]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f49d65630c44b4b epochNumber=2
{"provingJobId":"1:BASE_PARITY:2f49d65630c44b4b"}
21:21:33 [21:21:33.619]
INFO:
prover-client:proving-broker Proving Broker started
21:21:33 [21:21:33.619]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c05e4a05a9a6661a settled=false
{"provingJobId":"1:BASE_PARITY:c05e4a05a9a6661a","status":"pending"}
21:21:33 [21:21:33.620]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2f49d65630c44b4b settled=false
{"provingJobId":"1:BASE_PARITY:2f49d65630c44b4b","status":"pending"}
21:21:33 [21:21:33.620]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c05e4a05a9a6661a type=BASE_PARITY startedAt=1749849692568 older agent has taken job
{"provingJobId":"1:BASE_PARITY:c05e4a05a9a6661a"}
21:21:33 [21:21:33.621]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c05e4a05a9a6661a type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:c05e4a05a9a6661a"}
21:21:33 [21:21:33.621]
INFO:
prover-client:proving-broker Proving Broker started
21:21:33 [21:21:33.622]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d151bfaa89458ffb epochNumber=1
{"provingJobId":"1:BASE_PARITY:d151bfaa89458ffb"}
21:21:33 [21:21:33.622]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3224473fa6740ccb epochNumber=2
{"provingJobId":"1:BASE_PARITY:3224473fa6740ccb"}
21:21:34 [21:21:34.623]
INFO:
prover-client:proving-broker Proving Broker started
21:21:34 [21:21:34.624]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d151bfaa89458ffb settled=false
{"provingJobId":"1:BASE_PARITY:d151bfaa89458ffb","status":"pending"}
21:21:34 [21:21:34.624]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3224473fa6740ccb settled=false
{"provingJobId":"1:BASE_PARITY:3224473fa6740ccb","status":"pending"}
21:21:34 [21:21:34.625]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d151bfaa89458ffb type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:d151bfaa89458ffb"}
21:21:34 [21:21:34.626]
INFO:
prover-client:proving-broker Proving Broker started
21:21:34 [21:21:34.626]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35b907b60f4982c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:35b907b60f4982c6"}
21:21:34 [21:21:34.626]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79869c368d5c9519 epochNumber=2
{"provingJobId":"1:BASE_PARITY:79869c368d5c9519"}
21:21:35 [21:21:35.628]
INFO:
prover-client:proving-broker Proving Broker started
21:21:35 [21:21:35.628]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:35b907b60f4982c6 settled=false
{"provingJobId":"1:BASE_PARITY:35b907b60f4982c6","status":"pending"}
21:21:35 [21:21:35.628]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:79869c368d5c9519 settled=false
{"provingJobId":"1:BASE_PARITY:79869c368d5c9519","status":"pending"}
21:21:35 [21:21:35.629]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:35b907b60f4982c6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:35b907b60f4982c6"}
21:21:35 [21:21:35.629]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:35b907b60f4982c6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:35b907b60f4982c6"}
21:21:35 [21:21:35.630]
INFO:
prover-client:proving-broker Proving Broker started
21:21:35 [21:21:35.631]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:521cabb3037d78f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:521cabb3037d78f9"}
21:21:35 [21:21:35.631]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe63a2ae668df675 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fe63a2ae668df675"}
21:21:35 [21:21:35.631]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:521cabb3037d78f9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:521cabb3037d78f9"}
21:21:35 [21:21:35.632]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fe63a2ae668df675 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fe63a2ae668df675"}
21:21:35 [21:21:35.633]
INFO:
prover-client:proving-broker Proving Broker started
21:21:35 [21:21:35.633]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81b2fce4ae83bfa3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:81b2fce4ae83bfa3"}
21:21:35 [21:21:35.634]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:206dd84d0d73b01c epochNumber=2
{"provingJobId":"1:BASE_PARITY:206dd84d0d73b01c"}
21:21:35 [21:21:35.634]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:81b2fce4ae83bfa3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:81b2fce4ae83bfa3"}
21:21:35 [21:21:35.634]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:81b2fce4ae83bfa3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:81b2fce4ae83bfa3"}
21:21:35 [21:21:35.634]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:206dd84d0d73b01c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:206dd84d0d73b01c"}
21:21:35 [21:21:35.635]
INFO:
prover-client:proving-broker Proving Broker started
21:21:35 [21:21:35.636]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:69bdc31704899b67 err=test error
{"provingJoId":"1:BASE_PARITY:69bdc31704899b67"}
21:21:35 [21:21:35.637]
INFO:
prover-client:proving-broker Proving Broker started
21:21:35 [21:21:35.637]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5d6c94f13a84d398 not found
{"provingJobId":"1:BASE_PARITY:5d6c94f13a84d398"}
21:21:35 [21:21:35.638]
INFO:
prover-client:proving-broker Proving Broker started
21:21:35 [21:21:35.639]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e81a45dba38ff68 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2e81a45dba38ff68"}
21:21:35 [21:21:35.640]
INFO:
prover-client:proving-broker Proving Broker started
21:21:35 [21:21:35.640]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a61fcb7231980f0c epochNumber=1
{"provingJobId":"1:BASE_PARITY:a61fcb7231980f0c"}
21:21:35 [21:21:35.741]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a61fcb7231980f0c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a61fcb7231980f0c"}
21:21:35 [21:21:35.747]
INFO:
prover-client:proving-broker Proving Broker started
21:21:35 [21:21:35.748]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87120db1c39e5e62 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87120db1c39e5e62"}
21:21:35 [21:21:35.849]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:87120db1c39e5e62 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:87120db1c39e5e62"}
21:21:35 [21:21:35.856]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e86eb8042629410 epochNumber=10
{"provingJobId":"1:BASE_PARITY:6e86eb8042629410"}
21:21:35 [21:21:35.875]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:35 [21:21:35.982]
INFO:
prover-client:proving-broker Proving Broker started
21:21:35 [21:21:35.983]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7aacb8e4a2ab0dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7aacb8e4a2ab0dc"}
21:21:36 [21:21:36.134]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e7aacb8e4a2ab0dc timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e7aacb8e4a2ab0dc"}
21:21:36 [21:21:36.184]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.184]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:407a45a2afefac94 epochNumber=1
{"provingJobId":"1:BASE_PARITY:407a45a2afefac94"}
21:21:36 [21:21:36.185]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:407a45a2afefac94 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:407a45a2afefac94"}
21:21:36 [21:21:36.186]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.186]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:23ad6c4f9e4b9b54 epochNumber=1
{"provingJobId":"1:BASE_PARITY:23ad6c4f9e4b9b54"}
21:21:36 [21:21:36.187]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:23ad6c4f9e4b9b54 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:23ad6c4f9e4b9b54"}
21:21:36 [21:21:36.187]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:23ad6c4f9e4b9b54 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:23ad6c4f9e4b9b54"}
21:21:36 [21:21:36.188]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.189]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6901e844551d7ab9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6901e844551d7ab9"}
21:21:36 [21:21:36.189]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6901e844551d7ab9 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6901e844551d7ab9"}
21:21:36 [21:21:36.190]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.190]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5e3f18a3e8d06d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5e3f18a3e8d06d4"}
21:21:36 [21:21:36.191]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b5e3f18a3e8d06d4 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b5e3f18a3e8d06d4"}
21:21:36 [21:21:36.191]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a57584bfbfc7a961 epochNumber=3
{"provingJobId":"1:BASE_PARITY:a57584bfbfc7a961"}
21:21:36 [21:21:36.215]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:36 [21:21:36.216]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b5e3f18a3e8d06d4 err=test error
{"provingJoId":"1:BASE_PARITY:b5e3f18a3e8d06d4"}
21:21:36 [21:21:36.218]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.218]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:64ca7d53310bf598 settled=false
{"provingJobId":"1:BASE_PARITY:64ca7d53310bf598","status":"pending"}
21:21:36 [21:21:36.218]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ab3af42c7746875b settled=false
{"provingJobId":"1:BASE_PARITY:ab3af42c7746875b","status":"pending"}
21:21:36 [21:21:36.220]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.220]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e322bd43039b4046 settled=true
{"provingJobId":"1:BASE_PARITY:e322bd43039b4046","status":"fulfilled"}
21:21:36 [21:21:36.220]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:2a8fdf5575e0afd4 settled=true
{"provingJobId":"2:BASE_PARITY:2a8fdf5575e0afd4","status":"fulfilled"}
21:21:36 [21:21:36.221]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.222]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d7428f88845ec1d5 settled=true
{"provingJobId":"1:BASE_PARITY:d7428f88845ec1d5","status":"fulfilled"}
21:21:36 [21:21:36.222]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c0d7fcff4c5f7f2f settled=false
{"provingJobId":"1:BASE_PARITY:c0d7fcff4c5f7f2f","status":"pending"}
21:21:36 [21:21:36.223]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.224]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a5b18ef03d9fabe7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a5b18ef03d9fabe7"}
21:21:36 [21:21:36.226]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.226]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ffb353836c92386b epochNumber=1
{"provingJobId":"1:BASE_PARITY:ffb353836c92386b"}
21:21:36 [21:21:36.227]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:ffb353836c92386b: Error: db error: Error: db error
21:21:36 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:ffb353836c92386b"}
21:21:36 [21:21:36.228]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.228]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86c6de83642e3479 epochNumber=1
{"provingJobId":"1:BASE_PARITY:86c6de83642e3479"}
21:21:36 [21:21:36.228]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:86c6de83642e3479 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:86c6de83642e3479"}
21:21:36 [21:21:36.229]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:86c6de83642e3479 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:86c6de83642e3479"}
21:21:36 [21:21:36.230]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.230]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0566836c8e6e43c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0566836c8e6e43c2"}
21:21:36 [21:21:36.230]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0566836c8e6e43c2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0566836c8e6e43c2"}
21:21:36 [21:21:36.230]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0566836c8e6e43c2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0566836c8e6e43c2"}
21:21:36 [21:21:36.231]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:0566836c8e6e43c2: Error: db error
21:21:36 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:0566836c8e6e43c2"}
21:21:36 [21:21:36.232]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:745d678443d97e7b epochNumber=1
{"provingJobId":"1:BASE_PARITY:745d678443d97e7b"}
21:21:36 [21:21:36.232]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:745d678443d97e7b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:745d678443d97e7b"}
21:21:36 [21:21:36.233]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:745d678443d97e7b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:745d678443d97e7b"}
21:21:36 [21:21:36.233]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.234]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ec00079c0be7cb3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ec00079c0be7cb3"}
21:21:36 [21:21:36.234]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4ec00079c0be7cb3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4ec00079c0be7cb3"}
21:21:36 [21:21:36.234]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4ec00079c0be7cb3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:4ec00079c0be7cb3"}
21:21:36 [21:21:36.235]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:4ec00079c0be7cb3 jobErr=test error: Error: db error
21:21:36 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:4ec00079c0be7cb3"}
21:21:36 [21:21:36.236]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.236]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bf97ad60a76ec331 not found
{"provingJobId":"1:BASE_PARITY:bf97ad60a76ec331"}
21:21:36 [21:21:36.237]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.238]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3573b99886f681d0 err=test error
{"provingJoId":"1:BASE_PARITY:3573b99886f681d0"}
21:21:36 [21:21:36.239]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6252c22efc8ebe1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6252c22efc8ebe1"}
21:21:36 [21:21:36.250]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a6252c22efc8ebe1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a6252c22efc8ebe1"}
21:21:36 [21:21:36.250]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a6252c22efc8ebe1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a6252c22efc8ebe1"}
21:21:36 [21:21:36.260]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:fb279936431d9f2c epochNumber=2
{"provingJobId":"2:BASE_PARITY:fb279936431d9f2c"}
21:21:36 [21:21:36.260]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:fb279936431d9f2c type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:fb279936431d9f2c"}
21:21:36 [21:21:36.261]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:fb279936431d9f2c type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:fb279936431d9f2c"}
21:21:36 [21:21:36.321]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:d9e6d6f96518acc0 epochNumber=3
{"provingJobId":"3:BASE_PARITY:d9e6d6f96518acc0"}
21:21:36 [21:21:36.339]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:36 [21:21:36.381]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:d96fba2f2acd5cf7 epochNumber=4
{"provingJobId":"4:BASE_PARITY:d96fba2f2acd5cf7"}
21:21:36 [21:21:36.390]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:36 [21:21:36.443]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:fece8ee58c4fa96d epochNumber=5
{"provingJobId":"5:BASE_PARITY:fece8ee58c4fa96d"}
21:21:36 [21:21:36.465]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:36 [21:21:36.500]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6db3d3122fbbf512 epochNumber=42
{"provingJobId":"1:BASE_PARITY:6db3d3122fbbf512"}
21:21:36 [21:21:36.501]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testNlFl10/42 with map size 1073741824
21:21:36 [21:21:36.504]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.522]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:7f2bc11676827aa2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7f2bc11676827aa2"}
21:21:36 [21:21:36.526]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.526]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1b7ec985e2178a2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1b7ec985e2178a2"}
21:21:36 [21:21:36.527]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test40jkSh/1 with map size 1073741824
21:21:36 [21:21:36.530]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.551]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4470d90a9e023b8a epochNumber=1
{"provingJobId":"1:BASE_PARITY:4470d90a9e023b8a"}
21:21:36 [21:21:36.559]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.560]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3e8037e628874b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e3e8037e628874b6"}
21:21:36 [21:21:36.562]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQhtoVY/1 with map size 1073741824
21:21:36 [21:21:36.563]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.578]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e3e8037e628874b6 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e3e8037e628874b6"}
21:21:36 [21:21:36.581]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.581]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1da736e5d3704327 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1da736e5d3704327"}
21:21:36 [21:21:36.582]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTbGLz1/1 with map size 1073741824
21:21:36 [21:21:36.583]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.597]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1da736e5d3704327 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1da736e5d3704327"}
21:21:36 [21:21:36.597]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1da736e5d3704327 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1da736e5d3704327"}
21:21:36 [21:21:36.603]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1da736e5d3704327 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1da736e5d3704327"}
21:21:36 [21:21:36.605]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f226ea4a4d4fbb79 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f226ea4a4d4fbb79"}
21:21:36 [21:21:36.606]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKClr5M/1 with map size 1073741824
21:21:36 [21:21:36.607]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.625]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.627]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.628]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:336becc81f6b7fdd epochNumber=1
{"provingJobId":"1:BASE_PARITY:336becc81f6b7fdd"}
21:21:36 [21:21:36.628]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfOx14p/1 with map size 1073741824
21:21:36 [21:21:36.630]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.645]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:336becc81f6b7fdd
{"provingJobId":"1:BASE_PARITY:336becc81f6b7fdd"}
21:21:36 [21:21:36.645]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:336becc81f6b7fdd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:336becc81f6b7fdd"}
21:21:36 [21:21:36.645]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:336becc81f6b7fdd type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:336becc81f6b7fdd"}
21:21:36 [21:21:36.652]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.653]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1718d62bdc397330 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1718d62bdc397330"}
21:21:36 [21:21:36.653]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCo7CsK/1 with map size 1073741824
21:21:36 [21:21:36.654]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.668]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1718d62bdc397330
{"provingJobId":"1:BASE_PARITY:1718d62bdc397330"}
21:21:36 [21:21:36.669]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1718d62bdc397330 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1718d62bdc397330"}
21:21:36 [21:21:36.678]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.678]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc1068a8c8bbda25 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc1068a8c8bbda25"}
21:21:36 [21:21:36.681]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRkbvWf/1 with map size 1073741824
21:21:36 [21:21:36.682]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.704]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dc1068a8c8bbda25 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dc1068a8c8bbda25"}
21:21:36 [21:21:36.704]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dc1068a8c8bbda25 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dc1068a8c8bbda25"}
21:21:36 [21:21:36.711]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.712]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2efcbe1fb29e04f epochNumber=1
{"provingJobId":"1:BASE_PARITY:c2efcbe1fb29e04f"}
21:21:36 [21:21:36.714]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZ62mZh/1 with map size 1073741824
21:21:36 [21:21:36.715]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.736]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c2efcbe1fb29e04f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c2efcbe1fb29e04f"}
21:21:36 [21:21:36.737]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c2efcbe1fb29e04f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c2efcbe1fb29e04f"}
21:21:36 [21:21:36.744]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.744]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a0d8e517abe7714 epochNumber=0
{"provingJobId":"1:BASE_PARITY:1a0d8e517abe7714"}
21:21:36 [21:21:36.745]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test6v0XmB/0 with map size 1073741824
21:21:36 [21:21:36.746]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.760]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1a0d8e517abe7714 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1a0d8e517abe7714"}
21:21:36 [21:21:36.761]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42e26b81c73297ca epochNumber=0
{"provingJobId":"1:BASE_PARITY:42e26b81c73297ca"}
21:21:36 [21:21:36.761]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1a0d8e517abe7714 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1a0d8e517abe7714"}
21:21:36 [21:21:36.761]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5d6c64ede9e99271 epochNumber=0
{"provingJobId":"1:BASE_PARITY:5d6c64ede9e99271"}
21:21:36 [21:21:36.773]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.775]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d8a3bb0fb6d399b2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d8a3bb0fb6d399b2"}
21:21:36 [21:21:36.775]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMBlfoU/2 with map size 1073741824
21:21:36 [21:21:36.777]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e7b9c7f8bd5895b epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e7b9c7f8bd5895b"}
21:21:36 [21:21:36.791]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMBlfoU/1 with map size 1073741824
21:21:36 [21:21:36.792]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.808]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d15985f1dbc20579 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d15985f1dbc20579"}
21:21:36 [21:21:36.809]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testevPEhQ/1 with map size 1073741824
21:21:36 [21:21:36.810]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.828]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.828]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f4d008a97f6590b epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f4d008a97f6590b"}
21:21:36 [21:21:36.830]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTUGH5a/1 with map size 1073741824
21:21:36 [21:21:36.831]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:62664e0ef6b7af63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:62664e0ef6b7af63"}
21:21:36 [21:21:36.851]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1bf7b4641774b6d epochNumber=2
{"provingJobId":"1:BASE_PARITY:d1bf7b4641774b6d"}
21:21:36 [21:21:36.852]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTUGH5a/2 with map size 1073741824
21:21:36 [21:21:36.853]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.868]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ade04220fd4387c epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ade04220fd4387c"}
21:21:36 [21:21:36.875]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.875]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ed78093205d6723 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ed78093205d6723"}
21:21:36 [21:21:36.876]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6jvo2i/1 with map size 1073741824
21:21:36 [21:21:36.877]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.891]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4970dc254779a43 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4970dc254779a43"}
21:21:36 [21:21:36.896]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4700549fc987cf35 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4700549fc987cf35"}
21:21:36 [21:21:36.896]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6jvo2i/2 with map size 1073741824
21:21:36 [21:21:36.897]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.912]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a16c877ec524b2af epochNumber=1
{"provingJobId":"1:BASE_PARITY:a16c877ec524b2af"}
21:21:36 [21:21:36.920]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.920]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:185109ea2a3e3a22 epochNumber=1
{"provingJobId":"1:BASE_PARITY:185109ea2a3e3a22"}
21:21:36 [21:21:36.921]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCEnGZe/1 with map size 1073741824
21:21:36 [21:21:36.922]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.936]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81dcc52915a4b950 epochNumber=1
{"provingJobId":"1:BASE_PARITY:81dcc52915a4b950"}
21:21:36 [21:21:36.941]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be91c8b36973f9c7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:be91c8b36973f9c7"}
21:21:36 [21:21:36.942]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCEnGZe/2 with map size 1073741824
21:21:36 [21:21:36.943]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.957]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:516431d565998172 epochNumber=1
{"provingJobId":"1:BASE_PARITY:516431d565998172"}
21:21:36 [21:21:36.964]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.964]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04bccb004d136ae1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:04bccb004d136ae1"}
21:21:36 [21:21:36.965]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIPgaFZ/1 with map size 1073741824
21:21:36 [21:21:36.966]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:36 [21:21:36.980]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0868eff744ad466c epochNumber=1
{"provingJobId":"1:BASE_PARITY:0868eff744ad466c"}
21:21:36 [21:21:36.985]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:04bccb004d136ae1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:04bccb004d136ae1"}
21:21:36 [21:21:36.992]
INFO:
prover-client:proving-broker Proving Broker started
21:21:36 [21:21:36.992]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5937df096836e54d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5937df096836e54d"}
21:21:36 [21:21:36.993]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7PJvRn/1 with map size 1073741824
21:21:36 [21:21:36.994]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:37 [21:21:37.008]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13edc851c6511533 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13edc851c6511533"}
21:21:37 [21:21:37.012]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5937df096836e54d type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:5937df096836e54d"}
21:21:37 [21:21:37.020]
INFO:
prover-client:proving-broker Proving Broker started
21:21:37 [21:21:37.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c4370153ad6c01c epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c4370153ad6c01c"}
21:21:37 [21:21:37.021]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMU3TEI/1 with map size 1073741824
21:21:37 [21:21:37.022]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:37 [21:21:37.035]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a80cad83a7ed8503 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a80cad83a7ed8503"}
21:21:37 [21:21:37.040]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6c4370153ad6c01c type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:6c4370153ad6c01c"}
21:21:37 [21:21:37.042]
INFO:
prover-client:proving-broker Proving Broker started
21:21:37 [21:21:37.043]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4799547950314299 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4799547950314299"}
21:21:37 [21:21:37.043]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDkVEIC/1 with map size 1073741824
21:21:37 [21:21:37.044]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:37 [21:21:37.058]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4799547950314299
{"provingJobId":"1:BASE_PARITY:4799547950314299"}
21:21:37 [21:21:37.058]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4799547950314299 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4799547950314299"}
21:21:37 [21:21:37.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ffcd6b22fe0ba0aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:ffcd6b22fe0ba0aa"}
21:21:37 [21:21:37.067]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4799547950314299 has already been completed
{"provingJobId":"1:BASE_PARITY:4799547950314299"}
21:21:37 [21:21:37.070]
INFO:
prover-client:proving-broker Proving Broker started
21:21:37 [21:21:37.071]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ce541f321562a99 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0ce541f321562a99"}
21:21:37 [21:21:37.072]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6mOQzc/1 with map size 1073741824
21:21:37 [21:21:37.073]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:37 [21:21:37.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d9c8a391c02ec21 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1d9c8a391c02ec21"}
21:21:37 [21:21:37.087]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6mOQzc/2 with map size 1073741824
21:21:37 [21:21:37.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:38 [21:21:38.155]
INFO:
prover-client:proving-broker Proving Broker started
21:21:38 [21:21:38.159]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0ce541f321562a99 settled=false
{"provingJobId":"1:BASE_PARITY:0ce541f321562a99","status":"pending"}
21:21:38 [21:21:38.160]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1d9c8a391c02ec21 settled=false
{"provingJobId":"1:BASE_PARITY:1d9c8a391c02ec21","status":"pending"}
21:21:38 [21:21:38.161]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0ce541f321562a99 type=BASE_PARITY startedAt=1749849697102 older agent has taken job
{"provingJobId":"1:BASE_PARITY:0ce541f321562a99"}
21:21:38 [21:21:38.162]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0ce541f321562a99 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:0ce541f321562a99"}
21:21:38 [21:21:38.164]
INFO:
prover-client:proving-broker Proving Broker started
21:21:38 [21:21:38.165]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06d9c16c77ae61b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:06d9c16c77ae61b4"}
21:21:38 [21:21:38.166]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testM4B2zr/1 with map size 1073741824
21:21:38 [21:21:38.167]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:38 [21:21:38.183]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f0a16ff12405d9a epochNumber=2
{"provingJobId":"1:BASE_PARITY:4f0a16ff12405d9a"}
21:21:38 [21:21:38.184]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testM4B2zr/2 with map size 1073741824
21:21:38 [21:21:38.185]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:39 [21:21:39.199]
INFO:
prover-client:proving-broker Proving Broker started
21:21:39 [21:21:39.201]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:06d9c16c77ae61b4 settled=false
{"provingJobId":"1:BASE_PARITY:06d9c16c77ae61b4","status":"pending"}
21:21:39 [21:21:39.202]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4f0a16ff12405d9a settled=false
{"provingJobId":"1:BASE_PARITY:4f0a16ff12405d9a","status":"pending"}
21:21:39 [21:21:39.203]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:06d9c16c77ae61b4 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:06d9c16c77ae61b4"}
21:21:39 [21:21:39.206]
INFO:
prover-client:proving-broker Proving Broker started
21:21:39 [21:21:39.206]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1685b9f718996ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1685b9f718996ff"}
21:21:39 [21:21:39.207]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbasVlA/1 with map size 1073741824
21:21:39 [21:21:39.208]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:39 [21:21:39.225]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9455c0306611d31e epochNumber=2
{"provingJobId":"1:BASE_PARITY:9455c0306611d31e"}
21:21:39 [21:21:39.225]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testbasVlA/2 with map size 1073741824
21:21:39 [21:21:39.227]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.244]
INFO:
prover-client:proving-broker Proving Broker started
21:21:40 [21:21:40.246]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e1685b9f718996ff settled=false
{"provingJobId":"1:BASE_PARITY:e1685b9f718996ff","status":"pending"}
21:21:40 [21:21:40.248]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9455c0306611d31e settled=false
{"provingJobId":"1:BASE_PARITY:9455c0306611d31e","status":"pending"}
21:21:40 [21:21:40.249]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e1685b9f718996ff type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e1685b9f718996ff"}
21:21:40 [21:21:40.249]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e1685b9f718996ff type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e1685b9f718996ff"}
21:21:40 [21:21:40.259]
INFO:
prover-client:proving-broker Proving Broker started
21:21:40 [21:21:40.259]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dff222c2b9d12ae3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dff222c2b9d12ae3"}
21:21:40 [21:21:40.260]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZDUEcU/1 with map size 1073741824
21:21:40 [21:21:40.261]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.278]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0fdb3127b62d775 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c0fdb3127b62d775"}
21:21:40 [21:21:40.279]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZDUEcU/2 with map size 1073741824
21:21:40 [21:21:40.280]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.294]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:dff222c2b9d12ae3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:dff222c2b9d12ae3"}
21:21:40 [21:21:40.300]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c0fdb3127b62d775 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c0fdb3127b62d775"}
21:21:40 [21:21:40.307]
INFO:
prover-client:proving-broker Proving Broker started
21:21:40 [21:21:40.308]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4eda904d26a6eb19 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4eda904d26a6eb19"}
21:21:40 [21:21:40.308]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3QC9zb/1 with map size 1073741824
21:21:40 [21:21:40.309]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.324]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:23a13b730f7d1db6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:23a13b730f7d1db6"}
21:21:40 [21:21:40.324]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test3QC9zb/2 with map size 1073741824
21:21:40 [21:21:40.325]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.339]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4eda904d26a6eb19 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4eda904d26a6eb19"}
21:21:40 [21:21:40.339]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4eda904d26a6eb19 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4eda904d26a6eb19"}
21:21:40 [21:21:40.344]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:23a13b730f7d1db6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:23a13b730f7d1db6"}
21:21:40 [21:21:40.352]
INFO:
prover-client:proving-broker Proving Broker started
21:21:40 [21:21:40.353]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a45899f109a9be9d err=test error
{"provingJoId":"1:BASE_PARITY:a45899f109a9be9d"}
21:21:40 [21:21:40.356]
INFO:
prover-client:proving-broker Proving Broker started
21:21:40 [21:21:40.356]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:407b414d7083eff9 not found
{"provingJobId":"1:BASE_PARITY:407b414d7083eff9"}
21:21:40 [21:21:40.359]
INFO:
prover-client:proving-broker Proving Broker started
21:21:40 [21:21:40.360]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5e3d1c70d942c5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5e3d1c70d942c5e"}
21:21:40 [21:21:40.360]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgNP6Yx/1 with map size 1073741824
21:21:40 [21:21:40.361]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.378]
INFO:
prover-client:proving-broker Proving Broker started
21:21:40 [21:21:40.378]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df24286584b90f7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:df24286584b90f7d"}
21:21:40 [21:21:40.379]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPAKG4J/1 with map size 1073741824
21:21:40 [21:21:40.380]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.506]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:df24286584b90f7d timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:df24286584b90f7d"}
21:21:40 [21:21:40.510]
INFO:
prover-client:proving-broker Proving Broker started
21:21:40 [21:21:40.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10728946b32a7ca0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:10728946b32a7ca0"}
21:21:40 [21:21:40.511]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBlo4Rh/1 with map size 1073741824
21:21:40 [21:21:40.512]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.637]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:10728946b32a7ca0 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:10728946b32a7ca0"}
21:21:40 [21:21:40.643]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a26c67a2ebae74b5 epochNumber=10
{"provingJobId":"1:BASE_PARITY:a26c67a2ebae74b5"}
21:21:40 [21:21:40.644]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testBlo4Rh/10 with map size 1073741824
21:21:40 [21:21:40.645]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.687]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:40 [21:21:40.688]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:21:40 [21:21:40.790]
INFO:
prover-client:proving-broker Proving Broker started
21:21:40 [21:21:40.790]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9369110b1b2c4f35 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9369110b1b2c4f35"}
21:21:40 [21:21:40.791]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNXTXtl/1 with map size 1073741824
21:21:40 [21:21:40.792]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:40 [21:21:40.965]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9369110b1b2c4f35 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9369110b1b2c4f35"}
21:21:41 [21:21:41.010]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.011]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:38c80b28ab1a0e85 epochNumber=1
{"provingJobId":"1:BASE_PARITY:38c80b28ab1a0e85"}
21:21:41 [21:21:41.011]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3uXEY6/1 with map size 1073741824
21:21:41 [21:21:41.013]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.028]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:38c80b28ab1a0e85 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:38c80b28ab1a0e85"}
21:21:41 [21:21:41.030]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.031]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7da1fd2f7049102a epochNumber=1
{"provingJobId":"1:BASE_PARITY:7da1fd2f7049102a"}
21:21:41 [21:21:41.031]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgLr6q0/1 with map size 1073741824
21:21:41 [21:21:41.034]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.049]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:7da1fd2f7049102a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:7da1fd2f7049102a"}
21:21:41 [21:21:41.050]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7da1fd2f7049102a type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:7da1fd2f7049102a"}
21:21:41 [21:21:41.057]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.058]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ac3e09f28d1c5059 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ac3e09f28d1c5059"}
21:21:41 [21:21:41.058]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqdBsaS/1 with map size 1073741824
21:21:41 [21:21:41.060]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.074]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ac3e09f28d1c5059 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ac3e09f28d1c5059"}
21:21:41 [21:21:41.081]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bed9036398c3a36a epochNumber=1
{"provingJobId":"1:BASE_PARITY:bed9036398c3a36a"}
21:21:41 [21:21:41.082]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testo8cfCT/1 with map size 1073741824
21:21:41 [21:21:41.083]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.098]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bed9036398c3a36a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:bed9036398c3a36a"}
21:21:41 [21:21:41.099]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50500213c7e8bd98 epochNumber=3
{"provingJobId":"1:BASE_PARITY:50500213c7e8bd98"}
21:21:41 [21:21:41.100]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testo8cfCT/3 with map size 1073741824
21:21:41 [21:21:41.101]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.131]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:41 [21:21:41.131]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:21:41 [21:21:41.140]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bed9036398c3a36a err=test error
{"provingJoId":"1:BASE_PARITY:bed9036398c3a36a"}
21:21:41 [21:21:41.144]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrfMqAa/1 with map size 1073741824
21:21:41 [21:21:41.146]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.161]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrfMqAa/2 with map size 1073741824
21:21:41 [21:21:41.163]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.177]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.179]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:578af74264bdf9a2 settled=false
{"provingJobId":"1:BASE_PARITY:578af74264bdf9a2","status":"pending"}
21:21:41 [21:21:41.180]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f1055b392eb3d474 settled=false
{"provingJobId":"1:BASE_PARITY:f1055b392eb3d474","status":"pending"}
21:21:41 [21:21:41.184]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNJYOnp/1 with map size 1073741824
21:21:41 [21:21:41.186]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.200]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNJYOnp/2 with map size 1073741824
21:21:41 [21:21:41.202]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.226]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.228]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9592121b6852acbe settled=true
{"provingJobId":"1:BASE_PARITY:9592121b6852acbe","status":"fulfilled"}
21:21:41 [21:21:41.229]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:860d07a4352561dc settled=true
{"provingJobId":"2:BASE_PARITY:860d07a4352561dc","status":"fulfilled"}
21:21:41 [21:21:41.233]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOQjFyo/1 with map size 1073741824
21:21:41 [21:21:41.235]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.253]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOQjFyo/2 with map size 1073741824
21:21:41 [21:21:41.254]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.268]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.270]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e60ffe1ba8a65711 settled=true
{"provingJobId":"1:BASE_PARITY:e60ffe1ba8a65711","status":"fulfilled"}
21:21:41 [21:21:41.271]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:35802352e3a0e5de settled=false
{"provingJobId":"1:BASE_PARITY:35802352e3a0e5de","status":"pending"}
21:21:41 [21:21:41.275]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.275]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88c47c6fe4d0bcaf epochNumber=1
{"provingJobId":"1:BASE_PARITY:88c47c6fe4d0bcaf"}
21:21:41 [21:21:41.276]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfLCZVi/1 with map size 1073741824
21:21:41 [21:21:41.278]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.295]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.296]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4952f7f197b5b91 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e4952f7f197b5b91"}
21:21:41 [21:21:41.296]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:e4952f7f197b5b91: Error: db error: Error: db error
21:21:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:e4952f7f197b5b91"}
21:21:41 [21:21:41.299]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.299]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:725ffdea7b854943 epochNumber=1
{"provingJobId":"1:BASE_PARITY:725ffdea7b854943"}
21:21:41 [21:21:41.300]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXak4YY/1 with map size 1073741824
21:21:41 [21:21:41.301]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.315]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:725ffdea7b854943 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:725ffdea7b854943"}
21:21:41 [21:21:41.316]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:725ffdea7b854943 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:725ffdea7b854943"}
21:21:41 [21:21:41.323]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.324]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e4610883b913129 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e4610883b913129"}
21:21:41 [21:21:41.325]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuXaBRt/1 with map size 1073741824
21:21:41 [21:21:41.326]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.340]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4e4610883b913129 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4e4610883b913129"}
21:21:41 [21:21:41.341]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4e4610883b913129 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4e4610883b913129"}
21:21:41 [21:21:41.341]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:4e4610883b913129: Error: db error
21:21:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:4e4610883b913129"}
21:21:41 [21:21:41.346]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55f805ce3a3907a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:55f805ce3a3907a5"}
21:21:41 [21:21:41.347]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testme875t/1 with map size 1073741824
21:21:41 [21:21:41.348]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.363]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:55f805ce3a3907a5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:55f805ce3a3907a5"}
21:21:41 [21:21:41.364]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:55f805ce3a3907a5 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:55f805ce3a3907a5"}
21:21:41 [21:21:41.372]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.373]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b99f9579620589c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b99f9579620589c1"}
21:21:41 [21:21:41.374]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOn2hm3/1 with map size 1073741824
21:21:41 [21:21:41.375]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.406]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b99f9579620589c1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b99f9579620589c1"}
21:21:41 [21:21:41.406]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b99f9579620589c1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b99f9579620589c1"}
21:21:41 [21:21:41.407]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:b99f9579620589c1 jobErr=test error: Error: db error
21:21:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:b99f9579620589c1"}
21:21:41 [21:21:41.412]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.412]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e67c594d94828b8e not found
{"provingJobId":"1:BASE_PARITY:e67c594d94828b8e"}
21:21:41 [21:21:41.417]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.418]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0313674dd31371e4 err=test error
{"provingJoId":"1:BASE_PARITY:0313674dd31371e4"}
21:21:41 [21:21:41.422]
INFO:
prover-client:proving-broker Proving Broker started
21:21:41 [21:21:41.433]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:794ffe3cba896cc4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:794ffe3cba896cc4"}
21:21:41 [21:21:41.434]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiIj1UK/1 with map size 1073741824
21:21:41 [21:21:41.435]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.452]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:794ffe3cba896cc4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:794ffe3cba896cc4"}
21:21:41 [21:21:41.452]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:794ffe3cba896cc4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:794ffe3cba896cc4"}
21:21:41 [21:21:41.467]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:c32994da802a3672 epochNumber=2
{"provingJobId":"2:BASE_PARITY:c32994da802a3672"}
21:21:41 [21:21:41.468]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testiIj1UK/2 with map size 1073741824
21:21:41 [21:21:41.469]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.484]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:c32994da802a3672 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:c32994da802a3672"}
21:21:41 [21:21:41.484]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c32994da802a3672 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:c32994da802a3672"}
21:21:41 [21:21:41.550]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:a76ebe01f8a04f0b epochNumber=3
{"provingJobId":"3:BASE_PARITY:a76ebe01f8a04f0b"}
21:21:41 [21:21:41.551]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testiIj1UK/3 with map size 1073741824
21:21:41 [21:21:41.552]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.576]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:41 [21:21:41.576]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:21:41 [21:21:41.626]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:766961c388e492d3 epochNumber=4
{"provingJobId":"4:BASE_PARITY:766961c388e492d3"}
21:21:41 [21:21:41.627]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testiIj1UK/4 with map size 1073741824
21:21:41 [21:21:41.628]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.652]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:41 [21:21:41.653]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
21:21:41 [21:21:41.705]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:07054cc31a5f0429 epochNumber=5
{"provingJobId":"5:BASE_PARITY:07054cc31a5f0429"}
21:21:41 [21:21:41.706]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testiIj1UK/5 with map size 1073741824
21:21:41 [21:21:41.707]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:21:41 [21:21:41.730]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:21:41 [21:21:41.730]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
21:21:41
PASS src/proving_broker/proving_broker.test.ts (
10.94 s)
21:21:41 ProvingBroker
21:21:41 Producer API
21:21:41
✓ refuses stale jobs (33 ms)
21:21:41
✓ enqueues jobs (4 ms)
21:21:41
✓ ignores duplicate jobs (4 ms)
21:21:41
✓ reports correct status when enqueuing repeat jobs (4 ms)
21:21:41
✓ throws an error in case of duplicate job IDs (19 ms)
21:21:41
✓ returns not-found status for non-existing jobs (6 ms)
21:21:41
✓ cancels jobs in queue (2 ms)
21:21:41
✓ cancels jobs in-progress (2 ms)
21:21:41
✓ returns job result if successful (3 ms)
21:21:41
✓ returns job error if failed (2 ms)
21:21:41
✓ correctly returns job status for concurrent writes (3 ms)
21:21:41
✓ refuses stale jobs (29 ms)
21:21:41
✓ enqueues jobs (32 ms)
21:21:41
✓ ignores duplicate jobs (21 ms)
21:21:41
✓ reports correct status when enqueuing repeat jobs (24 ms)
21:21:41
✓ throws an error in case of duplicate job IDs (20 ms)
21:21:41
✓ returns not-found status for non-existing jobs (2 ms)
21:21:41
✓ cancels jobs in queue (25 ms)
21:21:41
✓ cancels jobs in-progress (24 ms)
21:21:41
✓ returns job result if successful (35 ms)
21:21:41
✓ returns job error if failed (33 ms)
21:21:41
✓ correctly returns job status for concurrent writes (29 ms)
21:21:41 Consumer API
21:21:41
✓ returns undefined if no jobs are available (1 ms)
21:21:41
✓ returns jobs in priority order (1 ms)
21:21:41
✓ returns undefined if no jobs are available for the given allowList (1 ms)
21:21:41
✓ returns a job if it is in the allowList (2 ms)
21:21:41
✓ returns the most important job if it is in the allowList (2 ms)
21:21:41
✓ returns any job if filter is empty (2 ms)
21:21:41
✓ returns a new job when reporting job success (2 ms)
21:21:41
✓ returns a new job when reporting permanent error (1 ms)
21:21:41
✓ returns a new job when reporting retry-able error (1 ms)
21:21:41
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
21:21:41
✓ returns a new job if job is already in progress elsewhere (1054 ms)
21:21:41
✓ avoids sending the same job to a new agent after a restart (1004 ms)
21:21:41
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
21:21:41
✓ tracks job result if in progress (2 ms)
21:21:41
✓ tracks job result even if job is in queue (2 ms)
21:21:41
✓ ignores reported job error if unknown job (1 ms)
21:21:41
✓ ignores job result if unknown job (2 ms)
21:21:41
✓ returns undefined if no jobs are available (2 ms)
21:21:41
✓ returns jobs in priority order (33 ms)
21:21:41
✓ returns undefined if no jobs are available for the given allowList (20 ms)
21:21:41
✓ returns a job if it is in the allowList (47 ms)
21:21:41
✓ returns the most important job if it is in the allowList (45 ms)
21:21:41
✓ returns any job if filter is empty (44 ms)
21:21:41
✓ returns a new job when reporting job success (27 ms)
21:21:41
✓ returns a new job when reporting permanent error (27 ms)
21:21:41
✓ returns a new job when reporting retry-able error (22 ms)
21:21:41
✓ returns a new job when reporting progress if current one is cancelled (27 ms)
21:21:41
✓ returns a new job if job is already in progress elsewhere (1094 ms)
21:21:41
✓ avoids sending the same job to a new agent after a restart (1041 ms)
21:21:41
✓ avoids sending a completed job to a new agent after a restart (1053 ms)
21:21:41
✓ tracks job result if in progress (48 ms)
21:21:41
✓ tracks job result even if job is in queue (45 ms)
21:21:41
✓ ignores reported job error if unknown job (3 ms)
21:21:41
✓ ignores job result if unknown job (3 ms)
21:21:41 Timeouts
21:21:41
✓ tracks in progress jobs (1 ms)
21:21:41
✓ re-enqueues jobs that time out (108 ms)
21:21:41
✓ cancel stale jobs that time out (235 ms)
21:21:41
✓ keeps the jobs in progress while it is alive (201 ms)
21:21:41
✓ tracks in progress jobs (18 ms)
21:21:41
✓ re-enqueues jobs that time out (131 ms)
21:21:41
✓ cancel stale jobs that time out (279 ms)
21:21:41
✓ keeps the jobs in progress while it is alive (218 ms)
21:21:41 Retries
21:21:41
✓ retries jobs (2 ms)
21:21:41
✓ retries up to a maximum number of times (2 ms)
21:21:41
✓ passing retry=false does not retry (1 ms)
21:21:41
✓ does not retry if job is stale (28 ms)
21:21:41
✓ retries jobs (21 ms)
21:21:41
✓ retries up to a maximum number of times (27 ms)
21:21:41
✓ passing retry=false does not retry (23 ms)
21:21:41
✓ does not retry if job is stale (62 ms)
21:21:41 Database management
21:21:41
✓ re-enqueues proof requests on start (2 ms)
21:21:41
✓ restores proof results on start (2 ms)
21:21:41
✓ only re-enqueues unfinished jobs (1 ms)
21:21:41
✓ saves job when enqueued (2 ms)
21:21:41
✓ does not retain job if database fails to save (2 ms)
21:21:41
✓ saves job result (2 ms)
21:21:41
✓ saves result even if database fails to save (2 ms)
21:21:41
✓ saves job error (2 ms)
21:21:41
✓ saves job error even if database fails to save (2 ms)
21:21:41
✓ does not save job result if job is unknown (2 ms)
21:21:41
✓ does not save job error if job is unknown (1 ms)
21:21:41
✓ cleans up old jobs periodically (257 ms)
21:21:41
✓ re-enqueues proof requests on start (39 ms)
21:21:41
✓ restores proof results on start (48 ms)
21:21:41
✓ only re-enqueues unfinished jobs (42 ms)
21:21:41
✓ saves job when enqueued (20 ms)
21:21:41
✓ does not retain job if database fails to save (3 ms)
21:21:41
✓ saves job result (24 ms)
21:21:41
✓ saves result even if database fails to save (22 ms)
21:21:41
✓ saves job error (27 ms)
21:21:41
✓ saves job error even if database fails to save (37 ms)
21:21:41
✓ does not save job result if job is unknown (4 ms)
21:21:41
✓ does not save job error if job is unknown (5 ms)
21:21:41
✓ cleans up old jobs periodically (354 ms)
21:21:41
21:21:41
Test Suites: 1 passed, 1 total
21:21:41
Tests: 96 passed, 96 total
21:21:41
Snapshots: 0 total
21:21:41
Time: 11.008 s
21:21:41
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
21:21:41
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?