Parent Log:
http://ci.aztec-labs.com/5c64f73239d8b492
Command: 3b26b88555004035 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/9cf4a24266245fa3ca003adb00e8ae759880c107
Env: REF_NAME=gh-readonly-queue/next/pr-14963-eeab628a4e44477cb6796c5a461ee0c0bb332d75 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 16 17:10:14 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14963_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
17:10:19 [17:10:19.159]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.165]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dcf548034bc75baf epochNumber=42
{"provingJobId":"1:BASE_PARITY:dcf548034bc75baf"}
17:10:19 [17:10:19.169]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:c149f8638992444d epochNumber=1
{"provingJobId":"1:BASE_PARITY:c149f8638992444d"}
17:10:19 [17:10:19.197]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.199]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:326adbc3d7bb1761 epochNumber=1
{"provingJobId":"1:BASE_PARITY:326adbc3d7bb1761"}
17:10:19 [17:10:19.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60a133692d36a596 epochNumber=1
{"provingJobId":"1:BASE_PARITY:60a133692d36a596"}
17:10:19 [17:10:19.202]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.202]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e06f932b477c2885 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e06f932b477c2885"}
17:10:19 [17:10:19.203]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e06f932b477c2885 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e06f932b477c2885"}
17:10:19 [17:10:19.205]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.205]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6fb6a938983c369 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6fb6a938983c369"}
17:10:19 [17:10:19.206]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c6fb6a938983c369 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c6fb6a938983c369"}
17:10:19 [17:10:19.207]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c6fb6a938983c369 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c6fb6a938983c369"}
17:10:19 [17:10:19.207]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c6fb6a938983c369 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c6fb6a938983c369"}
17:10:19 [17:10:19.209]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.209]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:62bba2a0313ff4b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:62bba2a0313ff4b5"}
17:10:19 [17:10:19.238]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.240]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.241]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9bd3c09e8f662f94 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9bd3c09e8f662f94"}
17:10:19 [17:10:19.242]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9bd3c09e8f662f94
{"provingJobId":"1:BASE_PARITY:9bd3c09e8f662f94"}
17:10:19 [17:10:19.242]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9bd3c09e8f662f94 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9bd3c09e8f662f94"}
17:10:19 [17:10:19.243]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9bd3c09e8f662f94 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:9bd3c09e8f662f94"}
17:10:19 [17:10:19.244]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.245]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a18e325cc9d3845 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a18e325cc9d3845"}
17:10:19 [17:10:19.246]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3a18e325cc9d3845
{"provingJobId":"1:BASE_PARITY:3a18e325cc9d3845"}
17:10:19 [17:10:19.246]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3a18e325cc9d3845 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3a18e325cc9d3845"}
17:10:19 [17:10:19.247]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.248]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f944d965f0342aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f944d965f0342aa"}
17:10:19 [17:10:19.248]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9f944d965f0342aa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9f944d965f0342aa"}
17:10:19 [17:10:19.248]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9f944d965f0342aa type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9f944d965f0342aa"}
17:10:19 [17:10:19.250]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.258]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:db578b06b3d59d8c epochNumber=1
{"provingJobId":"1:BASE_PARITY:db578b06b3d59d8c"}
17:10:19 [17:10:19.258]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:db578b06b3d59d8c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:db578b06b3d59d8c"}
17:10:19 [17:10:19.258]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:db578b06b3d59d8c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:db578b06b3d59d8c"}
17:10:19 [17:10:19.260]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.260]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d508d6d12f4a632 epochNumber=0
{"provingJobId":"1:BASE_PARITY:9d508d6d12f4a632"}
17:10:19 [17:10:19.260]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9d508d6d12f4a632 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9d508d6d12f4a632"}
17:10:19 [17:10:19.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70d47208f9b9a79e epochNumber=0
{"provingJobId":"1:BASE_PARITY:70d47208f9b9a79e"}
17:10:19 [17:10:19.261]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9d508d6d12f4a632 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9d508d6d12f4a632"}
17:10:19 [17:10:19.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82605faba7a33142 epochNumber=0
{"provingJobId":"1:BASE_PARITY:82605faba7a33142"}
17:10:19 [17:10:19.262]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.263]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.264]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfa1cb4e99f1d265 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dfa1cb4e99f1d265"}
17:10:19 [17:10:19.264]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cf43124a9cb72fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:0cf43124a9cb72fa"}
17:10:19 [17:10:19.265]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e092a50e59d5741 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2e092a50e59d5741"}
17:10:19 [17:10:19.266]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.267]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5ebe5880b56e1df epochNumber=1
{"provingJobId":"1:BASE_PARITY:f5ebe5880b56e1df"}
17:10:19 [17:10:19.267]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2591b89f3a33bbd0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2591b89f3a33bbd0"}
17:10:19 [17:10:19.267]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f18589f510ea7252 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f18589f510ea7252"}
17:10:19 [17:10:19.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4433caf8c2d85278 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4433caf8c2d85278"}
17:10:19 [17:10:19.268]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e778ded98f8b2d9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:e778ded98f8b2d9f"}
17:10:19 [17:10:19.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48c1177c2dc16342 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48c1177c2dc16342"}
17:10:19 [17:10:19.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c943c898e5d81ae0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c943c898e5d81ae0"}
17:10:19 [17:10:19.270]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0de17107278013dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:0de17107278013dc"}
17:10:19 [17:10:19.271]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.272]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48f667f1d182be47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48f667f1d182be47"}
17:10:19 [17:10:19.272]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1907b68888a6da01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1907b68888a6da01"}
17:10:19 [17:10:19.272]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b098f22e5c6b8b00 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b098f22e5c6b8b00"}
17:10:19 [17:10:19.272]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d058c5a00c81276 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d058c5a00c81276"}
17:10:19 [17:10:19.274]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.274]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:614e850f2a2c8e74 epochNumber=1
{"provingJobId":"1:BASE_PARITY:614e850f2a2c8e74"}
17:10:19 [17:10:19.275]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:046d53476f345520 epochNumber=1
{"provingJobId":"1:BASE_PARITY:046d53476f345520"}
17:10:19 [17:10:19.275]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:614e850f2a2c8e74 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:614e850f2a2c8e74"}
17:10:19 [17:10:19.276]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c079698f7c0bad3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c079698f7c0bad3"}
17:10:19 [17:10:19.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da0d96d8e72a1d6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:da0d96d8e72a1d6a"}
17:10:19 [17:10:19.277]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4c079698f7c0bad3 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:4c079698f7c0bad3"}
17:10:19 [17:10:19.278]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.278]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aebde8ce34d812f1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aebde8ce34d812f1"}
17:10:19 [17:10:19.279]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7e1192d906c34aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:d7e1192d906c34aa"}
17:10:19 [17:10:19.279]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:aebde8ce34d812f1 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:aebde8ce34d812f1"}
17:10:19 [17:10:19.280]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.280]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e41eb2fa1b59619 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e41eb2fa1b59619"}
17:10:19 [17:10:19.280]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4e41eb2fa1b59619
{"provingJobId":"1:BASE_PARITY:4e41eb2fa1b59619"}
17:10:19 [17:10:19.280]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4e41eb2fa1b59619 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:4e41eb2fa1b59619"}
17:10:19 [17:10:19.281]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a0010cd1a29faa1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6a0010cd1a29faa1"}
17:10:19 [17:10:19.281]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4e41eb2fa1b59619 has already been completed
{"provingJobId":"1:BASE_PARITY:4e41eb2fa1b59619"}
17:10:19 [17:10:19.282]
INFO:
prover-client:proving-broker Proving Broker started
17:10:19 [17:10:19.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bed39c0e9b969e38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bed39c0e9b969e38"}
17:10:19 [17:10:19.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34b4b33b7036178e epochNumber=2
{"provingJobId":"1:BASE_PARITY:34b4b33b7036178e"}
17:10:20 [17:10:20.334]
INFO:
prover-client:proving-broker Proving Broker started
17:10:20 [17:10:20.335]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bed39c0e9b969e38 settled=false
{"provingJobId":"1:BASE_PARITY:bed39c0e9b969e38","status":"pending"}
17:10:20 [17:10:20.336]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:34b4b33b7036178e settled=false
{"provingJobId":"1:BASE_PARITY:34b4b33b7036178e","status":"pending"}
17:10:20 [17:10:20.338]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bed39c0e9b969e38 type=BASE_PARITY startedAt=1750093819283 older agent has taken job
{"provingJobId":"1:BASE_PARITY:bed39c0e9b969e38"}
17:10:20 [17:10:20.338]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bed39c0e9b969e38 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:bed39c0e9b969e38"}
17:10:20 [17:10:20.340]
INFO:
prover-client:proving-broker Proving Broker started
17:10:20 [17:10:20.341]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0e97af7f7928a4b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0e97af7f7928a4b"}
17:10:20 [17:10:20.341]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ecc5931025ceda58 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ecc5931025ceda58"}
17:10:21 [17:10:21.342]
INFO:
prover-client:proving-broker Proving Broker started
17:10:21 [17:10:21.342]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e0e97af7f7928a4b settled=false
{"provingJobId":"1:BASE_PARITY:e0e97af7f7928a4b","status":"pending"}
17:10:21 [17:10:21.343]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ecc5931025ceda58 settled=false
{"provingJobId":"1:BASE_PARITY:ecc5931025ceda58","status":"pending"}
17:10:21 [17:10:21.344]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e0e97af7f7928a4b type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:e0e97af7f7928a4b"}
17:10:21 [17:10:21.346]
INFO:
prover-client:proving-broker Proving Broker started
17:10:21 [17:10:21.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:713ba2335d68e0b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:713ba2335d68e0b3"}
17:10:21 [17:10:21.347]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9aac4a63390fdd33 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9aac4a63390fdd33"}
17:10:22 [17:10:22.347]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.348]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:713ba2335d68e0b3 settled=false
{"provingJobId":"1:BASE_PARITY:713ba2335d68e0b3","status":"pending"}
17:10:22 [17:10:22.348]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9aac4a63390fdd33 settled=false
{"provingJobId":"1:BASE_PARITY:9aac4a63390fdd33","status":"pending"}
17:10:22 [17:10:22.349]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:713ba2335d68e0b3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:713ba2335d68e0b3"}
17:10:22 [17:10:22.350]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:713ba2335d68e0b3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:713ba2335d68e0b3"}
17:10:22 [17:10:22.352]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.353]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9688750baa4c2c5b epochNumber=1
{"provingJobId":"1:BASE_PARITY:9688750baa4c2c5b"}
17:10:22 [17:10:22.353]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fcf8640b7723834f epochNumber=2
{"provingJobId":"1:BASE_PARITY:fcf8640b7723834f"}
17:10:22 [17:10:22.354]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9688750baa4c2c5b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9688750baa4c2c5b"}
17:10:22 [17:10:22.354]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fcf8640b7723834f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fcf8640b7723834f"}
17:10:22 [17:10:22.356]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67cd37c8df474755 epochNumber=1
{"provingJobId":"1:BASE_PARITY:67cd37c8df474755"}
17:10:22 [17:10:22.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a1fb4f03ce6e618 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7a1fb4f03ce6e618"}
17:10:22 [17:10:22.357]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:67cd37c8df474755 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:67cd37c8df474755"}
17:10:22 [17:10:22.358]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:67cd37c8df474755 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:67cd37c8df474755"}
17:10:22 [17:10:22.358]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7a1fb4f03ce6e618 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7a1fb4f03ce6e618"}
17:10:22 [17:10:22.359]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.360]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:74352b124bf40720 err=test error
{"provingJoId":"1:BASE_PARITY:74352b124bf40720"}
17:10:22 [17:10:22.361]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.361]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:92a3d63d20005a42 not found
{"provingJobId":"1:BASE_PARITY:92a3d63d20005a42"}
17:10:22 [17:10:22.363]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.363]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f121846a11e2c99e epochNumber=1
{"provingJobId":"1:BASE_PARITY:f121846a11e2c99e"}
17:10:22 [17:10:22.365]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:deeeb2b51635c106 epochNumber=1
{"provingJobId":"1:BASE_PARITY:deeeb2b51635c106"}
17:10:22 [17:10:22.467]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:deeeb2b51635c106 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:deeeb2b51635c106"}
17:10:22 [17:10:22.474]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.474]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:003a7d0514c445af epochNumber=1
{"provingJobId":"1:BASE_PARITY:003a7d0514c445af"}
17:10:22 [17:10:22.576]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:003a7d0514c445af timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:003a7d0514c445af"}
17:10:22 [17:10:22.583]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d37d69cfafebd2c9 epochNumber=10
{"provingJobId":"1:BASE_PARITY:d37d69cfafebd2c9"}
17:10:22 [17:10:22.603]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:22 [17:10:22.710]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.710]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:797a94721c2e760c epochNumber=1
{"provingJobId":"1:BASE_PARITY:797a94721c2e760c"}
17:10:22 [17:10:22.863]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:797a94721c2e760c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:797a94721c2e760c"}
17:10:22 [17:10:22.914]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.914]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c469e5e1fa1cb93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c469e5e1fa1cb93"}
17:10:22 [17:10:22.915]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0c469e5e1fa1cb93 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0c469e5e1fa1cb93"}
17:10:22 [17:10:22.916]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.916]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5270e5b2b06bd8b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5270e5b2b06bd8b5"}
17:10:22 [17:10:22.917]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5270e5b2b06bd8b5 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:5270e5b2b06bd8b5"}
17:10:22 [17:10:22.918]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5270e5b2b06bd8b5 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:5270e5b2b06bd8b5"}
17:10:22 [17:10:22.918]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.919]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a38f1b61bd69d97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a38f1b61bd69d97"}
17:10:22 [17:10:22.919]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5a38f1b61bd69d97 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5a38f1b61bd69d97"}
17:10:22 [17:10:22.920]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.920]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ce5433a4b60b107 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ce5433a4b60b107"}
17:10:22 [17:10:22.921]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4ce5433a4b60b107 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:4ce5433a4b60b107"}
17:10:22 [17:10:22.921]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:124d8e83a519c569 epochNumber=3
{"provingJobId":"1:BASE_PARITY:124d8e83a519c569"}
17:10:22 [17:10:22.946]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:22 [17:10:22.947]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4ce5433a4b60b107 err=test error
{"provingJoId":"1:BASE_PARITY:4ce5433a4b60b107"}
17:10:22 [17:10:22.949]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.949]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:71e0cde62f3f63ee settled=false
{"provingJobId":"1:BASE_PARITY:71e0cde62f3f63ee","status":"pending"}
17:10:22 [17:10:22.950]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:888cbdbaab38acb2 settled=false
{"provingJobId":"1:BASE_PARITY:888cbdbaab38acb2","status":"pending"}
17:10:22 [17:10:22.953]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.953]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6b7e67aaf7d9803b settled=true
{"provingJobId":"1:BASE_PARITY:6b7e67aaf7d9803b","status":"fulfilled"}
17:10:22 [17:10:22.954]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:95e267b72aef83f1 settled=true
{"provingJobId":"2:BASE_PARITY:95e267b72aef83f1","status":"fulfilled"}
17:10:22 [17:10:22.955]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.955]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:62d4e1281f935c58 settled=true
{"provingJobId":"1:BASE_PARITY:62d4e1281f935c58","status":"fulfilled"}
17:10:22 [17:10:22.955]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:16ca8d08922cec85 settled=false
{"provingJobId":"1:BASE_PARITY:16ca8d08922cec85","status":"pending"}
17:10:22 [17:10:22.957]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.958]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d35f40f5a05aab2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d35f40f5a05aab2"}
17:10:22 [17:10:22.960]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.960]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86f2d4bf7dbd2c00 epochNumber=1
{"provingJobId":"1:BASE_PARITY:86f2d4bf7dbd2c00"}
17:10:22 [17:10:22.961]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:86f2d4bf7dbd2c00: Error: db error: Error: db error
17:10:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:86f2d4bf7dbd2c00"}
17:10:22 [17:10:22.962]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da39cb224c5b52aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:da39cb224c5b52aa"}
17:10:22 [17:10:22.963]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:da39cb224c5b52aa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:da39cb224c5b52aa"}
17:10:22 [17:10:22.963]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:da39cb224c5b52aa type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:da39cb224c5b52aa"}
17:10:22 [17:10:22.964]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.965]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd366b4db7ee91fc epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd366b4db7ee91fc"}
17:10:22 [17:10:22.965]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd366b4db7ee91fc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fd366b4db7ee91fc"}
17:10:22 [17:10:22.965]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fd366b4db7ee91fc type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fd366b4db7ee91fc"}
17:10:22 [17:10:22.966]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:fd366b4db7ee91fc: Error: db error
17:10:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:fd366b4db7ee91fc"}
17:10:22 [17:10:22.967]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.967]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d59b16df4bafd7c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d59b16df4bafd7c4"}
17:10:22 [17:10:22.968]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d59b16df4bafd7c4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d59b16df4bafd7c4"}
17:10:22 [17:10:22.968]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d59b16df4bafd7c4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d59b16df4bafd7c4"}
17:10:22 [17:10:22.969]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.970]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a0bc58bf7b4247e epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a0bc58bf7b4247e"}
17:10:22 [17:10:22.970]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3a0bc58bf7b4247e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3a0bc58bf7b4247e"}
17:10:22 [17:10:22.970]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3a0bc58bf7b4247e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3a0bc58bf7b4247e"}
17:10:22 [17:10:22.971]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:3a0bc58bf7b4247e jobErr=test error: Error: db error
17:10:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:3a0bc58bf7b4247e"}
17:10:22 [17:10:22.972]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.972]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ae9e87e9490e4254 not found
{"provingJobId":"1:BASE_PARITY:ae9e87e9490e4254"}
17:10:22 [17:10:22.974]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.974]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e377ad8d2459c98b err=test error
{"provingJoId":"1:BASE_PARITY:e377ad8d2459c98b"}
17:10:22 [17:10:22.975]
INFO:
prover-client:proving-broker Proving Broker started
17:10:22 [17:10:22.985]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b86a097e81f3772 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b86a097e81f3772"}
17:10:22 [17:10:22.986]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b86a097e81f3772 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7b86a097e81f3772"}
17:10:22 [17:10:22.986]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7b86a097e81f3772 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7b86a097e81f3772"}
17:10:22 [17:10:22.996]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:fb8d5fafe03ad6d8 epochNumber=2
{"provingJobId":"2:BASE_PARITY:fb8d5fafe03ad6d8"}
17:10:22 [17:10:22.997]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:fb8d5fafe03ad6d8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:fb8d5fafe03ad6d8"}
17:10:22 [17:10:22.997]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:fb8d5fafe03ad6d8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:fb8d5fafe03ad6d8"}
17:10:23 [17:10:23.058]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:4acaa850d1d0da6b epochNumber=3
{"provingJobId":"3:BASE_PARITY:4acaa850d1d0da6b"}
17:10:23 [17:10:23.076]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:23 [17:10:23.119]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:374ad636e1e079ec epochNumber=4
{"provingJobId":"4:BASE_PARITY:374ad636e1e079ec"}
17:10:23 [17:10:23.127]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:23 [17:10:23.183]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:a94f6904b39a8350 epochNumber=5
{"provingJobId":"5:BASE_PARITY:a94f6904b39a8350"}
17:10:23 [17:10:23.204]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:23 [17:10:23.237]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2685aec5ae9a5d32 epochNumber=42
{"provingJobId":"1:BASE_PARITY:2685aec5ae9a5d32"}
17:10:23 [17:10:23.240]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testEITVpC/42 with map size 1073741824
17:10:23 [17:10:23.243]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.270]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:7d99057d96c60873 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7d99057d96c60873"}
17:10:23 [17:10:23.276]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.277]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17910734dd4920fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:17910734dd4920fd"}
17:10:23 [17:10:23.277]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlMcYDM/1 with map size 1073741824
17:10:23 [17:10:23.279]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.298]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea86f430b3a77776 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea86f430b3a77776"}
17:10:23 [17:10:23.308]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.309]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b94a9ef38aefae2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b94a9ef38aefae2"}
17:10:23 [17:10:23.309]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQE64Tm/1 with map size 1073741824
17:10:23 [17:10:23.311]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.331]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4b94a9ef38aefae2 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4b94a9ef38aefae2"}
17:10:23 [17:10:23.335]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9a9265c71332062 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9a9265c71332062"}
17:10:23 [17:10:23.339]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvQlZFw/1 with map size 1073741824
17:10:23 [17:10:23.341]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.359]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b9a9265c71332062 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b9a9265c71332062"}
17:10:23 [17:10:23.360]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b9a9265c71332062 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b9a9265c71332062"}
17:10:23 [17:10:23.366]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b9a9265c71332062 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b9a9265c71332062"}
17:10:23 [17:10:23.370]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.371]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4c5ea3a514cfde7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4c5ea3a514cfde7"}
17:10:23 [17:10:23.372]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2rSe6O/1 with map size 1073741824
17:10:23 [17:10:23.374]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.398]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.404]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.405]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:afe0b16adb5ca002 epochNumber=1
{"provingJobId":"1:BASE_PARITY:afe0b16adb5ca002"}
17:10:23 [17:10:23.405]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmZKdBM/1 with map size 1073741824
17:10:23 [17:10:23.408]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.429]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:afe0b16adb5ca002
{"provingJobId":"1:BASE_PARITY:afe0b16adb5ca002"}
17:10:23 [17:10:23.430]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:afe0b16adb5ca002 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:afe0b16adb5ca002"}
17:10:23 [17:10:23.430]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:afe0b16adb5ca002 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:afe0b16adb5ca002"}
17:10:23 [17:10:23.437]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.438]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:011a75f2c5ff2fd9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:011a75f2c5ff2fd9"}
17:10:23 [17:10:23.438]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPzSmsj/1 with map size 1073741824
17:10:23 [17:10:23.440]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.450]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:011a75f2c5ff2fd9
{"provingJobId":"1:BASE_PARITY:011a75f2c5ff2fd9"}
17:10:23 [17:10:23.450]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:011a75f2c5ff2fd9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:011a75f2c5ff2fd9"}
17:10:23 [17:10:23.457]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.458]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:817b5fbc8dc2320f epochNumber=1
{"provingJobId":"1:BASE_PARITY:817b5fbc8dc2320f"}
17:10:23 [17:10:23.459]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1gwHCS/1 with map size 1073741824
17:10:23 [17:10:23.461]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.468]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:817b5fbc8dc2320f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:817b5fbc8dc2320f"}
17:10:23 [17:10:23.469]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:817b5fbc8dc2320f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:817b5fbc8dc2320f"}
17:10:23 [17:10:23.474]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.476]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e92735846dc12d80 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e92735846dc12d80"}
17:10:23 [17:10:23.477]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvMgjB5/1 with map size 1073741824
17:10:23 [17:10:23.479]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.489]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e92735846dc12d80 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e92735846dc12d80"}
17:10:23 [17:10:23.490]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e92735846dc12d80 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e92735846dc12d80"}
17:10:23 [17:10:23.495]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.496]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:daa5219de363dd78 epochNumber=0
{"provingJobId":"1:BASE_PARITY:daa5219de363dd78"}
17:10:23 [17:10:23.496]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testwGXeR2/0 with map size 1073741824
17:10:23 [17:10:23.498]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.511]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:daa5219de363dd78 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:daa5219de363dd78"}
17:10:23 [17:10:23.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78a83e732f824a38 epochNumber=0
{"provingJobId":"1:BASE_PARITY:78a83e732f824a38"}
17:10:23 [17:10:23.513]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:daa5219de363dd78 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:daa5219de363dd78"}
17:10:23 [17:10:23.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5279e64855825ffb epochNumber=0
{"provingJobId":"1:BASE_PARITY:5279e64855825ffb"}
17:10:23 [17:10:23.526]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.528]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.528]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df2c894f498ff8f6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:df2c894f498ff8f6"}
17:10:23 [17:10:23.529]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnvHXkO/2 with map size 1073741824
17:10:23 [17:10:23.530]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.548]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4215364668b68af1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4215364668b68af1"}
17:10:23 [17:10:23.549]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnvHXkO/1 with map size 1073741824
17:10:23 [17:10:23.551]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.569]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.570]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78b126041f10c60e epochNumber=1
{"provingJobId":"1:BASE_PARITY:78b126041f10c60e"}
17:10:23 [17:10:23.571]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testU9FttR/1 with map size 1073741824
17:10:23 [17:10:23.572]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.592]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.592]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc0610aeea29defe epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc0610aeea29defe"}
17:10:23 [17:10:23.593]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgtUkVl/1 with map size 1073741824
17:10:23 [17:10:23.594]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.610]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ad66dfa670018816 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ad66dfa670018816"}
17:10:23 [17:10:23.615]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2939f7764614b86 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b2939f7764614b86"}
17:10:23 [17:10:23.616]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testgtUkVl/2 with map size 1073741824
17:10:23 [17:10:23.618]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.635]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19ab94de0e993be7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:19ab94de0e993be7"}
17:10:23 [17:10:23.644]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.645]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6e23b60d74c5123 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d6e23b60d74c5123"}
17:10:23 [17:10:23.647]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrTyGiv/1 with map size 1073741824
17:10:23 [17:10:23.651]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.668]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eda182d1698847fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:eda182d1698847fa"}
17:10:23 [17:10:23.675]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb004f334ec19a7f epochNumber=2
{"provingJobId":"1:BASE_PARITY:eb004f334ec19a7f"}
17:10:23 [17:10:23.675]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrTyGiv/2 with map size 1073741824
17:10:23 [17:10:23.677]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b587e6341acd680f epochNumber=1
{"provingJobId":"1:BASE_PARITY:b587e6341acd680f"}
17:10:23 [17:10:23.701]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.702]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ac97ae9752670dd7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ac97ae9752670dd7"}
17:10:23 [17:10:23.703]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHAAvxk/1 with map size 1073741824
17:10:23 [17:10:23.704]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.721]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a7674794588dd841 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a7674794588dd841"}
17:10:23 [17:10:23.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13ca81264f79f230 epochNumber=2
{"provingJobId":"1:BASE_PARITY:13ca81264f79f230"}
17:10:23 [17:10:23.728]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testHAAvxk/2 with map size 1073741824
17:10:23 [17:10:23.730]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57feffe515acf411 epochNumber=1
{"provingJobId":"1:BASE_PARITY:57feffe515acf411"}
17:10:23 [17:10:23.784]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4312a313033dfa26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4312a313033dfa26"}
17:10:23 [17:10:23.788]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNdjFbc/1 with map size 1073741824
17:10:23 [17:10:23.790]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.809]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ef92790ebb1d8d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3ef92790ebb1d8d3"}
17:10:23 [17:10:23.815]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4312a313033dfa26 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4312a313033dfa26"}
17:10:23 [17:10:23.824]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.824]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7be3d9ee721d022c epochNumber=1
{"provingJobId":"1:BASE_PARITY:7be3d9ee721d022c"}
17:10:23 [17:10:23.825]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQIVcaB/1 with map size 1073741824
17:10:23 [17:10:23.828]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.851]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fe83219a6510a28 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fe83219a6510a28"}
17:10:23 [17:10:23.861]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7be3d9ee721d022c type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:7be3d9ee721d022c"}
17:10:23 [17:10:23.876]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.877]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:939fce6610977dfc epochNumber=1
{"provingJobId":"1:BASE_PARITY:939fce6610977dfc"}
17:10:23 [17:10:23.877]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVUbUDm/1 with map size 1073741824
17:10:23 [17:10:23.880]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.900]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b107bb88e36acef7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b107bb88e36acef7"}
17:10:23 [17:10:23.906]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:939fce6610977dfc type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:939fce6610977dfc"}
17:10:23 [17:10:23.909]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.910]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a068554d315b6a96 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a068554d315b6a96"}
17:10:23 [17:10:23.910]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZeaKkP/1 with map size 1073741824
17:10:23 [17:10:23.912]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.932]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a068554d315b6a96
{"provingJobId":"1:BASE_PARITY:a068554d315b6a96"}
17:10:23 [17:10:23.932]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a068554d315b6a96 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a068554d315b6a96"}
17:10:23 [17:10:23.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a544e16113c4f506 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a544e16113c4f506"}
17:10:23 [17:10:23.950]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a068554d315b6a96 has already been completed
{"provingJobId":"1:BASE_PARITY:a068554d315b6a96"}
17:10:23 [17:10:23.958]
INFO:
prover-client:proving-broker Proving Broker started
17:10:23 [17:10:23.960]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04e9015c2aba5c03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:04e9015c2aba5c03"}
17:10:23 [17:10:23.961]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFwURHZ/1 with map size 1073741824
17:10:23 [17:10:23.967]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:23 [17:10:23.989]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a75e5e48c38b988 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9a75e5e48c38b988"}
17:10:23 [17:10:23.991]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFwURHZ/2 with map size 1073741824
17:10:23 [17:10:23.996]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:25 [17:10:25.072]
INFO:
prover-client:proving-broker Proving Broker started
17:10:25 [17:10:25.077]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:04e9015c2aba5c03 settled=false
{"provingJobId":"1:BASE_PARITY:04e9015c2aba5c03","status":"pending"}
17:10:25 [17:10:25.079]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9a75e5e48c38b988 settled=false
{"provingJobId":"1:BASE_PARITY:9a75e5e48c38b988","status":"pending"}
17:10:25 [17:10:25.080]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:04e9015c2aba5c03 type=BASE_PARITY startedAt=1750093824020 older agent has taken job
{"provingJobId":"1:BASE_PARITY:04e9015c2aba5c03"}
17:10:25 [17:10:25.080]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:04e9015c2aba5c03 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:04e9015c2aba5c03"}
17:10:25 [17:10:25.085]
INFO:
prover-client:proving-broker Proving Broker started
17:10:25 [17:10:25.086]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82ee01d137ff7948 epochNumber=1
{"provingJobId":"1:BASE_PARITY:82ee01d137ff7948"}
17:10:25 [17:10:25.087]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlLsQ11/1 with map size 1073741824
17:10:25 [17:10:25.091]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:25 [17:10:25.112]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb9b0b6f4a0ca127 epochNumber=2
{"provingJobId":"1:BASE_PARITY:eb9b0b6f4a0ca127"}
17:10:25 [17:10:25.113]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlLsQ11/2 with map size 1073741824
17:10:25 [17:10:25.119]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:26 [17:10:26.137]
INFO:
prover-client:proving-broker Proving Broker started
17:10:26 [17:10:26.140]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:82ee01d137ff7948 settled=false
{"provingJobId":"1:BASE_PARITY:82ee01d137ff7948","status":"pending"}
17:10:26 [17:10:26.141]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:eb9b0b6f4a0ca127 settled=false
{"provingJobId":"1:BASE_PARITY:eb9b0b6f4a0ca127","status":"pending"}
17:10:26 [17:10:26.142]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:82ee01d137ff7948 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:82ee01d137ff7948"}
17:10:26 [17:10:26.147]
INFO:
prover-client:proving-broker Proving Broker started
17:10:26 [17:10:26.148]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be1d5856cfe952f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:be1d5856cfe952f9"}
17:10:26 [17:10:26.149]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2naW1Q/1 with map size 1073741824
17:10:26 [17:10:26.152]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:26 [17:10:26.173]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d72c06fbb12b7f2e epochNumber=2
{"provingJobId":"1:BASE_PARITY:d72c06fbb12b7f2e"}
17:10:26 [17:10:26.174]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2naW1Q/2 with map size 1073741824
17:10:26 [17:10:26.177]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.195]
INFO:
prover-client:proving-broker Proving Broker started
17:10:27 [17:10:27.197]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:be1d5856cfe952f9 settled=false
{"provingJobId":"1:BASE_PARITY:be1d5856cfe952f9","status":"pending"}
17:10:27 [17:10:27.199]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d72c06fbb12b7f2e settled=false
{"provingJobId":"1:BASE_PARITY:d72c06fbb12b7f2e","status":"pending"}
17:10:27 [17:10:27.200]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:be1d5856cfe952f9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:be1d5856cfe952f9"}
17:10:27 [17:10:27.200]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:be1d5856cfe952f9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:be1d5856cfe952f9"}
17:10:27 [17:10:27.211]
INFO:
prover-client:proving-broker Proving Broker started
17:10:27 [17:10:27.212]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fdcad0ec528f473a epochNumber=1
{"provingJobId":"1:BASE_PARITY:fdcad0ec528f473a"}
17:10:27 [17:10:27.212]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4MWY0J/1 with map size 1073741824
17:10:27 [17:10:27.214]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.230]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b33c3d11948443be epochNumber=2
{"provingJobId":"1:BASE_PARITY:b33c3d11948443be"}
17:10:27 [17:10:27.231]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4MWY0J/2 with map size 1073741824
17:10:27 [17:10:27.233]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.249]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fdcad0ec528f473a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fdcad0ec528f473a"}
17:10:27 [17:10:27.255]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b33c3d11948443be type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b33c3d11948443be"}
17:10:27 [17:10:27.263]
INFO:
prover-client:proving-broker Proving Broker started
17:10:27 [17:10:27.264]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a97f863ab8bf2a41 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a97f863ab8bf2a41"}
17:10:27 [17:10:27.265]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVhevWB/1 with map size 1073741824
17:10:27 [17:10:27.266]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.282]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2d596036ca7d720 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f2d596036ca7d720"}
17:10:27 [17:10:27.283]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVhevWB/2 with map size 1073741824
17:10:27 [17:10:27.285]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.301]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a97f863ab8bf2a41 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a97f863ab8bf2a41"}
17:10:27 [17:10:27.301]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a97f863ab8bf2a41 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a97f863ab8bf2a41"}
17:10:27 [17:10:27.307]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f2d596036ca7d720 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f2d596036ca7d720"}
17:10:27 [17:10:27.317]
INFO:
prover-client:proving-broker Proving Broker started
17:10:27 [17:10:27.318]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0e21dad4127626b4 err=test error
{"provingJoId":"1:BASE_PARITY:0e21dad4127626b4"}
17:10:27 [17:10:27.320]
INFO:
prover-client:proving-broker Proving Broker started
17:10:27 [17:10:27.321]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:101afaa84c0c4694 not found
{"provingJobId":"1:BASE_PARITY:101afaa84c0c4694"}
17:10:27 [17:10:27.323]
INFO:
prover-client:proving-broker Proving Broker started
17:10:27 [17:10:27.324]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3bc3dedc737baf89 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3bc3dedc737baf89"}
17:10:27 [17:10:27.324]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2z4czy/1 with map size 1073741824
17:10:27 [17:10:27.326]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.351]
INFO:
prover-client:proving-broker Proving Broker started
17:10:27 [17:10:27.352]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a7f4994d7253d045 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a7f4994d7253d045"}
17:10:27 [17:10:27.353]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQmPjCO/1 with map size 1073741824
17:10:27 [17:10:27.354]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.478]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a7f4994d7253d045 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a7f4994d7253d045"}
17:10:27 [17:10:27.488]
INFO:
prover-client:proving-broker Proving Broker started
17:10:27 [17:10:27.488]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2de9aecf62aad3a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d2de9aecf62aad3a"}
17:10:27 [17:10:27.489]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testswdzx6/1 with map size 1073741824
17:10:27 [17:10:27.492]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.639]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d2de9aecf62aad3a timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:d2de9aecf62aad3a"}
17:10:27 [17:10:27.642]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3967a734e90a127 epochNumber=10
{"provingJobId":"1:BASE_PARITY:e3967a734e90a127"}
17:10:27 [17:10:27.643]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testswdzx6/10 with map size 1073741824
17:10:27 [17:10:27.646]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.666]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:27 [17:10:27.666]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:10:27 [17:10:27.795]
INFO:
prover-client:proving-broker Proving Broker started
17:10:27 [17:10:27.796]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f4ef16df76b7df6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f4ef16df76b7df6"}
17:10:27 [17:10:27.797]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwEbZ1p/1 with map size 1073741824
17:10:27 [17:10:27.801]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:27 [17:10:27.975]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2f4ef16df76b7df6 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2f4ef16df76b7df6"}
17:10:28 [17:10:28.030]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.031]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43fa18c0ab108164 epochNumber=1
{"provingJobId":"1:BASE_PARITY:43fa18c0ab108164"}
17:10:28 [17:10:28.032]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnkXmAA/1 with map size 1073741824
17:10:28 [17:10:28.036]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.066]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:43fa18c0ab108164 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:43fa18c0ab108164"}
17:10:28 [17:10:28.073]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:efa53e2174db9563 epochNumber=1
{"provingJobId":"1:BASE_PARITY:efa53e2174db9563"}
17:10:28 [17:10:28.075]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testL5BJbO/1 with map size 1073741824
17:10:28 [17:10:28.078]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.100]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:efa53e2174db9563 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:efa53e2174db9563"}
17:10:28 [17:10:28.101]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:efa53e2174db9563 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:efa53e2174db9563"}
17:10:28 [17:10:28.112]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.112]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ba1058958b3e975 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ba1058958b3e975"}
17:10:28 [17:10:28.113]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkbbrQ1/1 with map size 1073741824
17:10:28 [17:10:28.115]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.135]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9ba1058958b3e975 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9ba1058958b3e975"}
17:10:28 [17:10:28.154]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.155]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be5d8a8ca361dfeb epochNumber=1
{"provingJobId":"1:BASE_PARITY:be5d8a8ca361dfeb"}
17:10:28 [17:10:28.156]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFYw3qx/1 with map size 1073741824
17:10:28 [17:10:28.162]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.181]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:be5d8a8ca361dfeb type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:be5d8a8ca361dfeb"}
17:10:28 [17:10:28.182]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6caf14a90e29c9fc epochNumber=3
{"provingJobId":"1:BASE_PARITY:6caf14a90e29c9fc"}
17:10:28 [17:10:28.183]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testFYw3qx/3 with map size 1073741824
17:10:28 [17:10:28.185]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.207]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:28 [17:10:28.207]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:10:28 [17:10:28.232]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:be5d8a8ca361dfeb err=test error
{"provingJoId":"1:BASE_PARITY:be5d8a8ca361dfeb"}
17:10:28 [17:10:28.239]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHgin0f/1 with map size 1073741824
17:10:28 [17:10:28.242]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.267]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testHgin0f/2 with map size 1073741824
17:10:28 [17:10:28.272]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.289]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.291]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bb7dc0237d940809 settled=false
{"provingJobId":"1:BASE_PARITY:bb7dc0237d940809","status":"pending"}
17:10:28 [17:10:28.292]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8c17cab86abf0e6b settled=false
{"provingJobId":"1:BASE_PARITY:8c17cab86abf0e6b","status":"pending"}
17:10:28 [17:10:28.297]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiCCBQS/1 with map size 1073741824
17:10:28 [17:10:28.299]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.321]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testiCCBQS/2 with map size 1073741824
17:10:28 [17:10:28.324]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.366]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.368]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0786d2526c125ff9 settled=true
{"provingJobId":"1:BASE_PARITY:0786d2526c125ff9","status":"fulfilled"}
17:10:28 [17:10:28.369]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:c8e017f77b8ded90 settled=true
{"provingJobId":"2:BASE_PARITY:c8e017f77b8ded90","status":"fulfilled"}
17:10:28 [17:10:28.378]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIE85yU/1 with map size 1073741824
17:10:28 [17:10:28.384]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.419]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIE85yU/2 with map size 1073741824
17:10:28 [17:10:28.422]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.448]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.455]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f119fe1fd993d77c settled=true
{"provingJobId":"1:BASE_PARITY:f119fe1fd993d77c","status":"fulfilled"}
17:10:28 [17:10:28.463]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:77e4d6d5191bd518 settled=false
{"provingJobId":"1:BASE_PARITY:77e4d6d5191bd518","status":"pending"}
17:10:28 [17:10:28.469]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.470]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:781f89c07b149910 epochNumber=1
{"provingJobId":"1:BASE_PARITY:781f89c07b149910"}
17:10:28 [17:10:28.472]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8p3Ocs/1 with map size 1073741824
17:10:28 [17:10:28.475]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.504]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf1bac3e66d70f6b epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf1bac3e66d70f6b"}
17:10:28 [17:10:28.507]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:bf1bac3e66d70f6b: Error: db error: Error: db error
17:10:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:bf1bac3e66d70f6b"}
17:10:28 [17:10:28.512]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:660da3a63e5ffcb0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:660da3a63e5ffcb0"}
17:10:28 [17:10:28.515]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVTRo15/1 with map size 1073741824
17:10:28 [17:10:28.518]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.537]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:660da3a63e5ffcb0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:660da3a63e5ffcb0"}
17:10:28 [17:10:28.538]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:660da3a63e5ffcb0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:660da3a63e5ffcb0"}
17:10:28 [17:10:28.562]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.563]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:732869bbb571c3aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:732869bbb571c3aa"}
17:10:28 [17:10:28.566]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1iGBKm/1 with map size 1073741824
17:10:28 [17:10:28.571]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.595]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:732869bbb571c3aa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:732869bbb571c3aa"}
17:10:28 [17:10:28.595]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:732869bbb571c3aa type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:732869bbb571c3aa"}
17:10:28 [17:10:28.596]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:732869bbb571c3aa: Error: db error
17:10:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:732869bbb571c3aa"}
17:10:28 [17:10:28.603]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.604]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5f8367a7f3f90a3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c5f8367a7f3f90a3"}
17:10:28 [17:10:28.606]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtrZdcc/1 with map size 1073741824
17:10:28 [17:10:28.611]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.633]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c5f8367a7f3f90a3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c5f8367a7f3f90a3"}
17:10:28 [17:10:28.633]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c5f8367a7f3f90a3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c5f8367a7f3f90a3"}
17:10:28 [17:10:28.646]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.646]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3055bf121727b5c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3055bf121727b5c5"}
17:10:28 [17:10:28.647]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhN7oBa/1 with map size 1073741824
17:10:28 [17:10:28.654]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.679]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3055bf121727b5c5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3055bf121727b5c5"}
17:10:28 [17:10:28.680]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3055bf121727b5c5 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3055bf121727b5c5"}
17:10:28 [17:10:28.680]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:3055bf121727b5c5 jobErr=test error: Error: db error
17:10:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:3055bf121727b5c5"}
17:10:28 [17:10:28.686]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.687]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0430818d064bc7f6 not found
{"provingJobId":"1:BASE_PARITY:0430818d064bc7f6"}
17:10:28 [17:10:28.693]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.694]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c43fe4e063be4afa err=test error
{"provingJoId":"1:BASE_PARITY:c43fe4e063be4afa"}
17:10:28 [17:10:28.702]
INFO:
prover-client:proving-broker Proving Broker started
17:10:28 [17:10:28.713]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3bad2265d28b77d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3bad2265d28b77d2"}
17:10:28 [17:10:28.715]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf33ABN/1 with map size 1073741824
17:10:28 [17:10:28.717]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.745]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3bad2265d28b77d2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3bad2265d28b77d2"}
17:10:28 [17:10:28.745]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3bad2265d28b77d2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3bad2265d28b77d2"}
17:10:28 [17:10:28.764]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:c01029cf3e582ad9 epochNumber=2
{"provingJobId":"2:BASE_PARITY:c01029cf3e582ad9"}
17:10:28 [17:10:28.765]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testf33ABN/2 with map size 1073741824
17:10:28 [17:10:28.770]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.790]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:c01029cf3e582ad9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:c01029cf3e582ad9"}
17:10:28 [17:10:28.791]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c01029cf3e582ad9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:c01029cf3e582ad9"}
17:10:28 [17:10:28.858]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:09e9ae7a6e7e03d0 epochNumber=3
{"provingJobId":"3:BASE_PARITY:09e9ae7a6e7e03d0"}
17:10:28 [17:10:28.860]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testf33ABN/3 with map size 1073741824
17:10:28 [17:10:28.869]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.908]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:28 [17:10:28.908]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:10:28 [17:10:28.953]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:f9da6b4aa525a238 epochNumber=4
{"provingJobId":"4:BASE_PARITY:f9da6b4aa525a238"}
17:10:28 [17:10:28.955]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testf33ABN/4 with map size 1073741824
17:10:28 [17:10:28.961]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:28 [17:10:28.988]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:28 [17:10:28.989]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
17:10:29 [17:10:29.043]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:01077e035059e0d9 epochNumber=5
{"provingJobId":"5:BASE_PARITY:01077e035059e0d9"}
17:10:29 [17:10:29.048]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testf33ABN/5 with map size 1073741824
17:10:29 [17:10:29.061]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:10:29 [17:10:29.096]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:10:29 [17:10:29.098]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
17:10:29
PASS src/proving_broker/proving_broker.test.ts (
13.459 s)
17:10:29 ProvingBroker
17:10:29 Producer API
17:10:29
✓ refuses stale jobs (39 ms)
17:10:29
✓ enqueues jobs (4 ms)
17:10:29
✓ ignores duplicate jobs (3 ms)
17:10:29
✓ reports correct status when enqueuing repeat jobs (3 ms)
17:10:29
✓ throws an error in case of duplicate job IDs (29 ms)
17:10:29
✓ returns not-found status for non-existing jobs (2 ms)
17:10:29
✓ cancels jobs in queue (3 ms)
17:10:29
✓ cancels jobs in-progress (2 ms)
17:10:29
✓ returns job result if successful (2 ms)
17:10:29
✓ returns job error if failed (10 ms)
17:10:29
✓ correctly returns job status for concurrent writes (3 ms)
17:10:29
✓ refuses stale jobs (39 ms)
17:10:29
✓ enqueues jobs (32 ms)
17:10:29
✓ ignores duplicate jobs (27 ms)
17:10:29
✓ reports correct status when enqueuing repeat jobs (35 ms)
17:10:29
✓ throws an error in case of duplicate job IDs (26 ms)
17:10:29
✓ returns not-found status for non-existing jobs (3 ms)
17:10:29
✓ cancels jobs in queue (35 ms)
17:10:29
✓ cancels jobs in-progress (20 ms)
17:10:29
✓ returns job result if successful (17 ms)
17:10:29
✓ returns job error if failed (21 ms)
17:10:29
✓ correctly returns job status for concurrent writes (31 ms)
17:10:29 Consumer API
17:10:29
✓ returns undefined if no jobs are available (1 ms)
17:10:29
✓ returns jobs in priority order (2 ms)
17:10:29
✓ returns undefined if no jobs are available for the given allowList (1 ms)
17:10:29
✓ returns a job if it is in the allowList (2 ms)
17:10:29
✓ returns the most important job if it is in the allowList (2 ms)
17:10:29
✓ returns any job if filter is empty (2 ms)
17:10:29
✓ returns a new job when reporting job success (3 ms)
17:10:29
✓ returns a new job when reporting permanent error (2 ms)
17:10:29
✓ returns a new job when reporting retry-able error (1 ms)
17:10:29
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
17:10:29
✓ returns a new job if job is already in progress elsewhere (1057 ms)
17:10:29
✓ avoids sending the same job to a new agent after a restart (1006 ms)
17:10:29
✓ avoids sending a completed job to a new agent after a restart (1006 ms)
17:10:29
✓ tracks job result if in progress (4 ms)
17:10:29
✓ tracks job result even if job is in queue (4 ms)
17:10:29
✓ ignores reported job error if unknown job (1 ms)
17:10:29
✓ ignores job result if unknown job (2 ms)
17:10:29
✓ returns undefined if no jobs are available (2 ms)
17:10:29
✓ returns jobs in priority order (41 ms)
17:10:29
✓ returns undefined if no jobs are available for the given allowList (22 ms)
17:10:29
✓ returns a job if it is in the allowList (51 ms)
17:10:29
✓ returns the most important job if it is in the allowList (58 ms)
17:10:29
✓ returns any job if filter is empty (82 ms)
17:10:29
✓ returns a new job when reporting job success (40 ms)
17:10:29
✓ returns a new job when reporting permanent error (50 ms)
17:10:29
✓ returns a new job when reporting retry-able error (34 ms)
17:10:29
✓ returns a new job when reporting progress if current one is cancelled (46 ms)
17:10:29
✓ returns a new job if job is already in progress elsewhere (1129 ms)
17:10:29
✓ avoids sending the same job to a new agent after a restart (1062 ms)
17:10:29
✓ avoids sending a completed job to a new agent after a restart (1063 ms)
17:10:29
✓ tracks job result if in progress (53 ms)
17:10:29
✓ tracks job result even if job is in queue (53 ms)
17:10:29
✓ ignores reported job error if unknown job (3 ms)
17:10:29
✓ ignores job result if unknown job (2 ms)
17:10:29 Timeouts
17:10:29
✓ tracks in progress jobs (2 ms)
17:10:29
✓ re-enqueues jobs that time out (109 ms)
17:10:29
✓ cancel stale jobs that time out (236 ms)
17:10:29
✓ keeps the jobs in progress while it is alive (204 ms)
17:10:29
✓ tracks in progress jobs (27 ms)
17:10:29
✓ re-enqueues jobs that time out (137 ms)
17:10:29
✓ cancel stale jobs that time out (307 ms)
17:10:29
✓ keeps the jobs in progress while it is alive (233 ms)
17:10:29 Retries
17:10:29
✓ retries jobs (3 ms)
17:10:29
✓ retries up to a maximum number of times (2 ms)
17:10:29
✓ passing retry=false does not retry (2 ms)
17:10:29
✓ does not retry if job is stale (28 ms)
17:10:29
✓ retries jobs (41 ms)
17:10:29
✓ retries up to a maximum number of times (40 ms)
17:10:29
✓ passing retry=false does not retry (38 ms)
17:10:29
✓ does not retry if job is stale (86 ms)
17:10:29 Database management
17:10:29
✓ re-enqueues proof requests on start (4 ms)
17:10:29
✓ restores proof results on start (2 ms)
17:10:29
✓ only re-enqueues unfinished jobs (1 ms)
17:10:29
✓ saves job when enqueued (3 ms)
17:10:29
✓ does not retain job if database fails to save (3 ms)
17:10:29
✓ saves job result (2 ms)
17:10:29
✓ saves result even if database fails to save (2 ms)
17:10:29
✓ saves job error (2 ms)
17:10:29
✓ saves job error even if database fails to save (2 ms)
17:10:29
✓ does not save job result if job is unknown (2 ms)
17:10:29
✓ does not save job error if job is unknown (2 ms)
17:10:29
✓ cleans up old jobs periodically (259 ms)
17:10:29
✓ re-enqueues proof requests on start (59 ms)
17:10:29
✓ restores proof results on start (77 ms)
17:10:29
✓ only re-enqueues unfinished jobs (95 ms)
17:10:29
✓ saves job when enqueued (32 ms)
17:10:29
✓ does not retain job if database fails to save (8 ms)
17:10:29
✓ saves job result (42 ms)
17:10:29
✓ saves result even if database fails to save (47 ms)
17:10:29
✓ saves job error (43 ms)
17:10:29
✓ saves job error even if database fails to save (39 ms)
17:10:29
✓ does not save job result if job is unknown (5 ms)
17:10:29
✓ does not save job error if job is unknown (8 ms)
17:10:29
✓ cleans up old jobs periodically (440 ms)
17:10:29
17:10:29
Test Suites: 1 passed, 1 total
17:10:29
Tests: 96 passed, 96 total
17:10:29
Snapshots: 0 total
17:10:29
Time: 13.58 s
17:10:29
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
17:10:29
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?