Parent Log:
http://ci.aztec-labs.com/5d2b04153c5165af
Command: 9e675ed54321cc5a yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/96a426245731b3f48fcb6dd5e2a683c858706602
Env: REF_NAME=gh-readonly-queue/next/pr-15083-5867520ed258adecfee692b3694d693b2c7964ca CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 16 21:53:40 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15083_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
21:53:44 [21:53:44.676]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.683]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82b387efbd35e5d3 epochNumber=42
{"provingJobId":"1:BASE_PARITY:82b387efbd35e5d3"}
21:53:44 [21:53:44.687]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:ddd3694abf7f1cc6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ddd3694abf7f1cc6"}
21:53:44 [21:53:44.725]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.725]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb10af1c37a31f19 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb10af1c37a31f19"}
21:53:44 [21:53:44.726]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:190cc24d495df5bf epochNumber=1
{"provingJobId":"1:BASE_PARITY:190cc24d495df5bf"}
21:53:44 [21:53:44.728]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.728]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec2be5b4467a4328 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec2be5b4467a4328"}
21:53:44 [21:53:44.729]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ec2be5b4467a4328 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ec2be5b4467a4328"}
21:53:44 [21:53:44.731]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.732]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e318e04f5f720394 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e318e04f5f720394"}
21:53:44 [21:53:44.732]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e318e04f5f720394 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e318e04f5f720394"}
21:53:44 [21:53:44.733]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e318e04f5f720394 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e318e04f5f720394"}
21:53:44 [21:53:44.733]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e318e04f5f720394 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e318e04f5f720394"}
21:53:44 [21:53:44.734]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.735]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c375bfc823724e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c375bfc823724e0"}
21:53:44 [21:53:44.756]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.758]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.759]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d82c7d7269f170e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d82c7d7269f170e8"}
21:53:44 [21:53:44.759]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d82c7d7269f170e8
{"provingJobId":"1:BASE_PARITY:d82c7d7269f170e8"}
21:53:44 [21:53:44.760]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d82c7d7269f170e8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d82c7d7269f170e8"}
21:53:44 [21:53:44.760]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d82c7d7269f170e8 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:d82c7d7269f170e8"}
21:53:44 [21:53:44.761]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5461a496b0157e7c epochNumber=1
{"provingJobId":"1:BASE_PARITY:5461a496b0157e7c"}
21:53:44 [21:53:44.762]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5461a496b0157e7c
{"provingJobId":"1:BASE_PARITY:5461a496b0157e7c"}
21:53:44 [21:53:44.763]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5461a496b0157e7c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5461a496b0157e7c"}
21:53:44 [21:53:44.764]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.764]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b3ffc424f5cb159b epochNumber=1
{"provingJobId":"1:BASE_PARITY:b3ffc424f5cb159b"}
21:53:44 [21:53:44.765]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b3ffc424f5cb159b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b3ffc424f5cb159b"}
21:53:44 [21:53:44.765]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b3ffc424f5cb159b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b3ffc424f5cb159b"}
21:53:44 [21:53:44.766]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.767]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bceeed04211a3bbf epochNumber=1
{"provingJobId":"1:BASE_PARITY:bceeed04211a3bbf"}
21:53:44 [21:53:44.767]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bceeed04211a3bbf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bceeed04211a3bbf"}
21:53:44 [21:53:44.767]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bceeed04211a3bbf type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bceeed04211a3bbf"}
21:53:44 [21:53:44.768]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b01e0c7df8269ef6 epochNumber=0
{"provingJobId":"1:BASE_PARITY:b01e0c7df8269ef6"}
21:53:44 [21:53:44.769]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b01e0c7df8269ef6 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b01e0c7df8269ef6"}
21:53:44 [21:53:44.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:62fbd859b99b155a epochNumber=0
{"provingJobId":"1:BASE_PARITY:62fbd859b99b155a"}
21:53:44 [21:53:44.769]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b01e0c7df8269ef6 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b01e0c7df8269ef6"}
21:53:44 [21:53:44.770]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c624e2d1cab71f7a epochNumber=0
{"provingJobId":"1:BASE_PARITY:c624e2d1cab71f7a"}
21:53:44 [21:53:44.770]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.771]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:111d09ac3a4526bf epochNumber=2
{"provingJobId":"1:BASE_PARITY:111d09ac3a4526bf"}
21:53:44 [21:53:44.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4232a59985a20eea epochNumber=1
{"provingJobId":"1:BASE_PARITY:4232a59985a20eea"}
21:53:44 [21:53:44.773]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.773]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:061e3fa89ad91a52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:061e3fa89ad91a52"}
21:53:44 [21:53:44.774]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d5b41a67172ef61 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d5b41a67172ef61"}
21:53:44 [21:53:44.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fffe81fd39a3dc1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fffe81fd39a3dc1"}
21:53:44 [21:53:44.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5da0a6eb309c1cf0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5da0a6eb309c1cf0"}
21:53:44 [21:53:44.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e7d84e47dfeb8f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e7d84e47dfeb8f9"}
21:53:44 [21:53:44.776]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3957ddc963a0604 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e3957ddc963a0604"}
21:53:44 [21:53:44.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7d058d5509e80366 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7d058d5509e80366"}
21:53:44 [21:53:44.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:422eccd127ad3900 epochNumber=2
{"provingJobId":"1:BASE_PARITY:422eccd127ad3900"}
21:53:44 [21:53:44.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c3f733a53f20ebd epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c3f733a53f20ebd"}
21:53:44 [21:53:44.778]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.779]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1690ebcd2e4fd65a epochNumber=1
{"provingJobId":"1:BASE_PARITY:1690ebcd2e4fd65a"}
21:53:44 [21:53:44.779]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5884e1f16d788edf epochNumber=1
{"provingJobId":"1:BASE_PARITY:5884e1f16d788edf"}
21:53:44 [21:53:44.779]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:806186558a97cb61 epochNumber=2
{"provingJobId":"1:BASE_PARITY:806186558a97cb61"}
21:53:44 [21:53:44.779]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5962ea8b3e23413c epochNumber=1
{"provingJobId":"1:BASE_PARITY:5962ea8b3e23413c"}
21:53:44 [21:53:44.780]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.780]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87f82ac75721aa1c epochNumber=1
{"provingJobId":"1:BASE_PARITY:87f82ac75721aa1c"}
21:53:44 [21:53:44.781]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56bb7d4c04b08b39 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56bb7d4c04b08b39"}
21:53:44 [21:53:44.781]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:87f82ac75721aa1c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:87f82ac75721aa1c"}
21:53:44 [21:53:44.782]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:efeb8318a5e4378b epochNumber=1
{"provingJobId":"1:BASE_PARITY:efeb8318a5e4378b"}
21:53:44 [21:53:44.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56705cc44f668882 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56705cc44f668882"}
21:53:44 [21:53:44.783]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:efeb8318a5e4378b type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:efeb8318a5e4378b"}
21:53:44 [21:53:44.784]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.784]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f35c7051654584fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:f35c7051654584fd"}
21:53:44 [21:53:44.784]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:424e66a2b67f1c98 epochNumber=1
{"provingJobId":"1:BASE_PARITY:424e66a2b67f1c98"}
21:53:44 [21:53:44.785]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f35c7051654584fd type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:f35c7051654584fd"}
21:53:44 [21:53:44.786]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:930d4a1cb49623e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:930d4a1cb49623e3"}
21:53:44 [21:53:44.787]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:930d4a1cb49623e3
{"provingJobId":"1:BASE_PARITY:930d4a1cb49623e3"}
21:53:44 [21:53:44.787]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:930d4a1cb49623e3 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:930d4a1cb49623e3"}
21:53:44 [21:53:44.787]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:581a31d6ed36a06b epochNumber=1
{"provingJobId":"1:BASE_PARITY:581a31d6ed36a06b"}
21:53:44 [21:53:44.788]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:930d4a1cb49623e3 has already been completed
{"provingJobId":"1:BASE_PARITY:930d4a1cb49623e3"}
21:53:44 [21:53:44.802]
INFO:
prover-client:proving-broker Proving Broker started
21:53:44 [21:53:44.803]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79fd29e6f4b9cbaa epochNumber=1
{"provingJobId":"1:BASE_PARITY:79fd29e6f4b9cbaa"}
21:53:44 [21:53:44.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d7298a4dda84e59 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9d7298a4dda84e59"}
21:53:45 [21:53:45.856]
INFO:
prover-client:proving-broker Proving Broker started
21:53:45 [21:53:45.858]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:79fd29e6f4b9cbaa settled=false
{"provingJobId":"1:BASE_PARITY:79fd29e6f4b9cbaa","status":"pending"}
21:53:45 [21:53:45.858]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9d7298a4dda84e59 settled=false
{"provingJobId":"1:BASE_PARITY:9d7298a4dda84e59","status":"pending"}
21:53:45 [21:53:45.859]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:79fd29e6f4b9cbaa type=BASE_PARITY startedAt=1750110824804 older agent has taken job
{"provingJobId":"1:BASE_PARITY:79fd29e6f4b9cbaa"}
21:53:45 [21:53:45.859]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:79fd29e6f4b9cbaa type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:79fd29e6f4b9cbaa"}
21:53:45 [21:53:45.861]
INFO:
prover-client:proving-broker Proving Broker started
21:53:45 [21:53:45.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57a8585bfc209f04 epochNumber=1
{"provingJobId":"1:BASE_PARITY:57a8585bfc209f04"}
21:53:45 [21:53:45.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58855825f3b7932f epochNumber=2
{"provingJobId":"1:BASE_PARITY:58855825f3b7932f"}
21:53:46 [21:53:46.863]
INFO:
prover-client:proving-broker Proving Broker started
21:53:46 [21:53:46.863]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:57a8585bfc209f04 settled=false
{"provingJobId":"1:BASE_PARITY:57a8585bfc209f04","status":"pending"}
21:53:46 [21:53:46.863]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:58855825f3b7932f settled=false
{"provingJobId":"1:BASE_PARITY:58855825f3b7932f","status":"pending"}
21:53:46 [21:53:46.864]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:57a8585bfc209f04 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:57a8585bfc209f04"}
21:53:46 [21:53:46.865]
INFO:
prover-client:proving-broker Proving Broker started
21:53:46 [21:53:46.866]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c85a05dad74b9912 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c85a05dad74b9912"}
21:53:46 [21:53:46.866]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e680b7f1733a711 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2e680b7f1733a711"}
21:53:47 [21:53:47.867]
INFO:
prover-client:proving-broker Proving Broker started
21:53:47 [21:53:47.868]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c85a05dad74b9912 settled=false
{"provingJobId":"1:BASE_PARITY:c85a05dad74b9912","status":"pending"}
21:53:47 [21:53:47.868]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2e680b7f1733a711 settled=false
{"provingJobId":"1:BASE_PARITY:2e680b7f1733a711","status":"pending"}
21:53:47 [21:53:47.869]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c85a05dad74b9912 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c85a05dad74b9912"}
21:53:47 [21:53:47.869]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c85a05dad74b9912 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c85a05dad74b9912"}
21:53:47 [21:53:47.871]
INFO:
prover-client:proving-broker Proving Broker started
21:53:47 [21:53:47.872]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cba2f8091c1b4ae6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cba2f8091c1b4ae6"}
21:53:47 [21:53:47.872]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c95ed5ff30ebb72 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0c95ed5ff30ebb72"}
21:53:47 [21:53:47.873]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cba2f8091c1b4ae6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cba2f8091c1b4ae6"}
21:53:47 [21:53:47.874]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0c95ed5ff30ebb72 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0c95ed5ff30ebb72"}
21:53:47 [21:53:47.875]
INFO:
prover-client:proving-broker Proving Broker started
21:53:47 [21:53:47.875]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5aea5ad4ec0af23b epochNumber=1
{"provingJobId":"1:BASE_PARITY:5aea5ad4ec0af23b"}
21:53:47 [21:53:47.876]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00ba1adc9846c7c7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:00ba1adc9846c7c7"}
21:53:47 [21:53:47.876]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5aea5ad4ec0af23b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5aea5ad4ec0af23b"}
21:53:47 [21:53:47.876]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5aea5ad4ec0af23b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5aea5ad4ec0af23b"}
21:53:47 [21:53:47.877]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:00ba1adc9846c7c7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:00ba1adc9846c7c7"}
21:53:47 [21:53:47.878]
INFO:
prover-client:proving-broker Proving Broker started
21:53:47 [21:53:47.878]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:dac22389b640a676 err=test error
{"provingJoId":"1:BASE_PARITY:dac22389b640a676"}
21:53:47 [21:53:47.879]
INFO:
prover-client:proving-broker Proving Broker started
21:53:47 [21:53:47.880]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d2279ce8143edb51 not found
{"provingJobId":"1:BASE_PARITY:d2279ce8143edb51"}
21:53:47 [21:53:47.881]
INFO:
prover-client:proving-broker Proving Broker started
21:53:47 [21:53:47.882]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:db861bd1c133dd7a epochNumber=1
{"provingJobId":"1:BASE_PARITY:db861bd1c133dd7a"}
21:53:47 [21:53:47.883]
INFO:
prover-client:proving-broker Proving Broker started
21:53:47 [21:53:47.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d382a38c21f6b03f epochNumber=1
{"provingJobId":"1:BASE_PARITY:d382a38c21f6b03f"}
21:53:47 [21:53:47.985]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d382a38c21f6b03f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:d382a38c21f6b03f"}
21:53:47 [21:53:47.992]
INFO:
prover-client:proving-broker Proving Broker started
21:53:47 [21:53:47.993]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a01e85761800cd2a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a01e85761800cd2a"}
21:53:48 [21:53:48.094]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a01e85761800cd2a timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a01e85761800cd2a"}
21:53:48 [21:53:48.096]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c96269fce200f63 epochNumber=10
{"provingJobId":"1:BASE_PARITY:3c96269fce200f63"}
21:53:48 [21:53:48.121]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:48 [21:53:48.224]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.225]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8a1fdfaaac0ed6b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8a1fdfaaac0ed6b4"}
21:53:48 [21:53:48.378]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8a1fdfaaac0ed6b4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8a1fdfaaac0ed6b4"}
21:53:48 [21:53:48.428]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.429]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af71cadd99b0bcab epochNumber=1
{"provingJobId":"1:BASE_PARITY:af71cadd99b0bcab"}
21:53:48 [21:53:48.430]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:af71cadd99b0bcab type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:af71cadd99b0bcab"}
21:53:48 [21:53:48.431]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.431]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79ea6c057a790b3a epochNumber=1
{"provingJobId":"1:BASE_PARITY:79ea6c057a790b3a"}
21:53:48 [21:53:48.432]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:79ea6c057a790b3a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:79ea6c057a790b3a"}
21:53:48 [21:53:48.433]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:79ea6c057a790b3a type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:79ea6c057a790b3a"}
21:53:48 [21:53:48.434]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.435]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:33bfe8bcb394aa0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:33bfe8bcb394aa0e"}
21:53:48 [21:53:48.435]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:33bfe8bcb394aa0e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:33bfe8bcb394aa0e"}
21:53:48 [21:53:48.436]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6cabf5efeff0ef19 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6cabf5efeff0ef19"}
21:53:48 [21:53:48.438]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6cabf5efeff0ef19 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6cabf5efeff0ef19"}
21:53:48 [21:53:48.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:489ff5b3b50400c3 epochNumber=3
{"provingJobId":"1:BASE_PARITY:489ff5b3b50400c3"}
21:53:48 [21:53:48.463]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:48 [21:53:48.465]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6cabf5efeff0ef19 err=test error
{"provingJoId":"1:BASE_PARITY:6cabf5efeff0ef19"}
21:53:48 [21:53:48.467]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.468]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c066b26839793d88 settled=false
{"provingJobId":"1:BASE_PARITY:c066b26839793d88","status":"pending"}
21:53:48 [21:53:48.468]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:52c44ec7389b434d settled=false
{"provingJobId":"1:BASE_PARITY:52c44ec7389b434d","status":"pending"}
21:53:48 [21:53:48.470]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.470]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5bac9c8fb6a74d9b settled=true
{"provingJobId":"1:BASE_PARITY:5bac9c8fb6a74d9b","status":"fulfilled"}
21:53:48 [21:53:48.470]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:aed2272ac27c32c9 settled=true
{"provingJobId":"2:BASE_PARITY:aed2272ac27c32c9","status":"fulfilled"}
21:53:48 [21:53:48.471]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.471]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:db332607e88b4d4e settled=true
{"provingJobId":"1:BASE_PARITY:db332607e88b4d4e","status":"fulfilled"}
21:53:48 [21:53:48.472]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0a66f962bccf05e0 settled=false
{"provingJobId":"1:BASE_PARITY:0a66f962bccf05e0","status":"pending"}
21:53:48 [21:53:48.473]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.474]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:702cbb66db3e2a96 epochNumber=1
{"provingJobId":"1:BASE_PARITY:702cbb66db3e2a96"}
21:53:48 [21:53:48.475]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.476]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46f1bc8e7d578f86 epochNumber=1
{"provingJobId":"1:BASE_PARITY:46f1bc8e7d578f86"}
21:53:48 [21:53:48.476]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:46f1bc8e7d578f86: Error: db error: Error: db error
21:53:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:46f1bc8e7d578f86"}
21:53:48 [21:53:48.477]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d718a0707d4a4fda epochNumber=1
{"provingJobId":"1:BASE_PARITY:d718a0707d4a4fda"}
21:53:48 [21:53:48.478]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d718a0707d4a4fda type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d718a0707d4a4fda"}
21:53:48 [21:53:48.478]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d718a0707d4a4fda type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d718a0707d4a4fda"}
21:53:48 [21:53:48.479]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.479]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:358f842a44e47f8e epochNumber=1
{"provingJobId":"1:BASE_PARITY:358f842a44e47f8e"}
21:53:48 [21:53:48.480]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:358f842a44e47f8e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:358f842a44e47f8e"}
21:53:48 [21:53:48.480]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:358f842a44e47f8e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:358f842a44e47f8e"}
21:53:48 [21:53:48.481]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:358f842a44e47f8e: Error: db error
21:53:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:358f842a44e47f8e"}
21:53:48 [21:53:48.481]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2816e0e9b1baafd epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2816e0e9b1baafd"}
21:53:48 [21:53:48.482]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f2816e0e9b1baafd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f2816e0e9b1baafd"}
21:53:48 [21:53:48.482]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f2816e0e9b1baafd type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f2816e0e9b1baafd"}
21:53:48 [21:53:48.484]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6dfcfa5b4fc7c7d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6dfcfa5b4fc7c7d3"}
21:53:48 [21:53:48.484]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6dfcfa5b4fc7c7d3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6dfcfa5b4fc7c7d3"}
21:53:48 [21:53:48.485]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6dfcfa5b4fc7c7d3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6dfcfa5b4fc7c7d3"}
21:53:48 [21:53:48.485]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:6dfcfa5b4fc7c7d3 jobErr=test error: Error: db error
21:53:48 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:6dfcfa5b4fc7c7d3"}
21:53:48 [21:53:48.486]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.487]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:29f809f8dead63a4 not found
{"provingJobId":"1:BASE_PARITY:29f809f8dead63a4"}
21:53:48 [21:53:48.488]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.488]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:aed8cea6eb1d094f err=test error
{"provingJoId":"1:BASE_PARITY:aed8cea6eb1d094f"}
21:53:48 [21:53:48.489]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c26eda0bd5939f45 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c26eda0bd5939f45"}
21:53:48 [21:53:48.500]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c26eda0bd5939f45 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c26eda0bd5939f45"}
21:53:48 [21:53:48.501]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c26eda0bd5939f45 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c26eda0bd5939f45"}
21:53:48 [21:53:48.512]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:eb3ba31cb7170ead epochNumber=2
{"provingJobId":"2:BASE_PARITY:eb3ba31cb7170ead"}
21:53:48 [21:53:48.513]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:eb3ba31cb7170ead type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:eb3ba31cb7170ead"}
21:53:48 [21:53:48.513]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:eb3ba31cb7170ead type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:eb3ba31cb7170ead"}
21:53:48 [21:53:48.574]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:107fe1758c396c27 epochNumber=3
{"provingJobId":"3:BASE_PARITY:107fe1758c396c27"}
21:53:48 [21:53:48.590]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:48 [21:53:48.635]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:9074b41da37c4c4b epochNumber=4
{"provingJobId":"4:BASE_PARITY:9074b41da37c4c4b"}
21:53:48 [21:53:48.641]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:48 [21:53:48.698]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:383843fda90eabab epochNumber=5
{"provingJobId":"5:BASE_PARITY:383843fda90eabab"}
21:53:48 [21:53:48.717]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:48 [21:53:48.753]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.753]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:189e3265176e4c92 epochNumber=42
{"provingJobId":"1:BASE_PARITY:189e3265176e4c92"}
21:53:48 [21:53:48.758]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testFDpW5D/42 with map size 1073741824
21:53:48 [21:53:48.767]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:48 [21:53:48.791]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:348288053a3ab637 epochNumber=1
{"provingJobId":"1:BASE_PARITY:348288053a3ab637"}
21:53:48 [21:53:48.801]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22c96860a95503a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:22c96860a95503a5"}
21:53:48 [21:53:48.803]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrwxNnc/1 with map size 1073741824
21:53:48 [21:53:48.805]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:48 [21:53:48.829]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54f869dc2e05f2b2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:54f869dc2e05f2b2"}
21:53:48 [21:53:48.845]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c3eb46824f8e297 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c3eb46824f8e297"}
21:53:48 [21:53:48.847]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1anN0E/1 with map size 1073741824
21:53:48 [21:53:48.850]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:48 [21:53:48.872]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0c3eb46824f8e297 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0c3eb46824f8e297"}
21:53:48 [21:53:48.877]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.877]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8cd128a86f6205e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8cd128a86f6205e6"}
21:53:48 [21:53:48.878]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbJHuEI/1 with map size 1073741824
21:53:48 [21:53:48.881]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:48 [21:53:48.914]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8cd128a86f6205e6 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8cd128a86f6205e6"}
21:53:48 [21:53:48.915]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8cd128a86f6205e6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8cd128a86f6205e6"}
21:53:48 [21:53:48.922]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8cd128a86f6205e6 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8cd128a86f6205e6"}
21:53:48 [21:53:48.926]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.926]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e3bdbb500711aa59 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e3bdbb500711aa59"}
21:53:48 [21:53:48.927]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyZm5LP/1 with map size 1073741824
21:53:48 [21:53:48.929]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:48 [21:53:48.952]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.955]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.955]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f70cfee2bb2e6622 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f70cfee2bb2e6622"}
21:53:48 [21:53:48.956]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKR5hQU/1 with map size 1073741824
21:53:48 [21:53:48.958]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:48 [21:53:48.976]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f70cfee2bb2e6622
{"provingJobId":"1:BASE_PARITY:f70cfee2bb2e6622"}
21:53:48 [21:53:48.977]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f70cfee2bb2e6622 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f70cfee2bb2e6622"}
21:53:48 [21:53:48.977]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f70cfee2bb2e6622 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f70cfee2bb2e6622"}
21:53:48 [21:53:48.988]
INFO:
prover-client:proving-broker Proving Broker started
21:53:48 [21:53:48.989]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:966a04eb0a67f166 epochNumber=1
{"provingJobId":"1:BASE_PARITY:966a04eb0a67f166"}
21:53:48 [21:53:48.989]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9P45v3/1 with map size 1073741824
21:53:48 [21:53:48.995]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.025]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:966a04eb0a67f166
{"provingJobId":"1:BASE_PARITY:966a04eb0a67f166"}
21:53:49 [21:53:49.026]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:966a04eb0a67f166 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:966a04eb0a67f166"}
21:53:49 [21:53:49.036]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.038]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b477eaeb0c3483c epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b477eaeb0c3483c"}
21:53:49 [21:53:49.039]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlplZZL/1 with map size 1073741824
21:53:49 [21:53:49.042]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.061]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7b477eaeb0c3483c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7b477eaeb0c3483c"}
21:53:49 [21:53:49.061]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7b477eaeb0c3483c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7b477eaeb0c3483c"}
21:53:49 [21:53:49.071]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.072]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0843eb84cc00c036 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0843eb84cc00c036"}
21:53:49 [21:53:49.073]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxWA9c1/1 with map size 1073741824
21:53:49 [21:53:49.075]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.095]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0843eb84cc00c036 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0843eb84cc00c036"}
21:53:49 [21:53:49.095]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0843eb84cc00c036 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0843eb84cc00c036"}
21:53:49 [21:53:49.109]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.110]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5de3b676f8c4dcc0 epochNumber=0
{"provingJobId":"1:BASE_PARITY:5de3b676f8c4dcc0"}
21:53:49 [21:53:49.111]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testPnx6O8/0 with map size 1073741824
21:53:49 [21:53:49.113]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.132]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5de3b676f8c4dcc0 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5de3b676f8c4dcc0"}
21:53:49 [21:53:49.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:317776cf56542295 epochNumber=0
{"provingJobId":"1:BASE_PARITY:317776cf56542295"}
21:53:49 [21:53:49.134]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5de3b676f8c4dcc0 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5de3b676f8c4dcc0"}
21:53:49 [21:53:49.135]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c20e4c33d593fb77 epochNumber=0
{"provingJobId":"1:BASE_PARITY:c20e4c33d593fb77"}
21:53:49 [21:53:49.149]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.151]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.152]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f55c1a5eab12cc7d epochNumber=2
{"provingJobId":"1:BASE_PARITY:f55c1a5eab12cc7d"}
21:53:49 [21:53:49.154]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyzxvd7/2 with map size 1073741824
21:53:49 [21:53:49.156]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.174]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1141b822418df5c epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1141b822418df5c"}
21:53:49 [21:53:49.175]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyzxvd7/1 with map size 1073741824
21:53:49 [21:53:49.177]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.203]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.204]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a41b6afaa559820 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a41b6afaa559820"}
21:53:49 [21:53:49.206]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyxncLw/1 with map size 1073741824
21:53:49 [21:53:49.210]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.234]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e11e5c260dc89f7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e11e5c260dc89f7"}
21:53:49 [21:53:49.236]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testX8gv1n/1 with map size 1073741824
21:53:49 [21:53:49.239]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.260]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb3bdd3e0a84aeee epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb3bdd3e0a84aeee"}
21:53:49 [21:53:49.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dced4077bd3421e0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dced4077bd3421e0"}
21:53:49 [21:53:49.267]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testX8gv1n/2 with map size 1073741824
21:53:49 [21:53:49.269]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.288]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ca156921091caec epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ca156921091caec"}
21:53:49 [21:53:49.298]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.299]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe50a59df6e54725 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe50a59df6e54725"}
21:53:49 [21:53:49.299]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGs69W5/1 with map size 1073741824
21:53:49 [21:53:49.303]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.322]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:18efc79207914127 epochNumber=1
{"provingJobId":"1:BASE_PARITY:18efc79207914127"}
21:53:49 [21:53:49.330]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d4b17c836e16f5c epochNumber=2
{"provingJobId":"1:BASE_PARITY:6d4b17c836e16f5c"}
21:53:49 [21:53:49.331]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGs69W5/2 with map size 1073741824
21:53:49 [21:53:49.333]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.349]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e58286ccb2bb5643 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e58286ccb2bb5643"}
21:53:49 [21:53:49.358]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.359]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef1342b35c449cd1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef1342b35c449cd1"}
21:53:49 [21:53:49.361]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testljBk4g/1 with map size 1073741824
21:53:49 [21:53:49.363]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.379]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:657fa09673137c06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:657fa09673137c06"}
21:53:49 [21:53:49.386]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c2478e4396686d1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0c2478e4396686d1"}
21:53:49 [21:53:49.387]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testljBk4g/2 with map size 1073741824
21:53:49 [21:53:49.389]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.410]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1235c8007b98f80a epochNumber=1
{"provingJobId":"1:BASE_PARITY:1235c8007b98f80a"}
21:53:49 [21:53:49.420]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.421]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51ff5d14a76af4f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:51ff5d14a76af4f3"}
21:53:49 [21:53:49.421]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIIesEK/1 with map size 1073741824
21:53:49 [21:53:49.423]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.441]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ac72b09d697962af epochNumber=1
{"provingJobId":"1:BASE_PARITY:ac72b09d697962af"}
21:53:49 [21:53:49.447]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:51ff5d14a76af4f3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:51ff5d14a76af4f3"}
21:53:49 [21:53:49.456]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.457]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a71610ad9954b541 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a71610ad9954b541"}
21:53:49 [21:53:49.457]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKcqlgW/1 with map size 1073741824
21:53:49 [21:53:49.459]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.477]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c0cd0f56977d76d epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c0cd0f56977d76d"}
21:53:49 [21:53:49.484]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a71610ad9954b541 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:a71610ad9954b541"}
21:53:49 [21:53:49.493]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.494]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f22ca81a05ebf5c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f22ca81a05ebf5c9"}
21:53:49 [21:53:49.495]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrXcCVC/1 with map size 1073741824
21:53:49 [21:53:49.500]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.523]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8dbd40e1f47ca9d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8dbd40e1f47ca9d5"}
21:53:49 [21:53:49.532]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f22ca81a05ebf5c9 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:f22ca81a05ebf5c9"}
21:53:49 [21:53:49.541]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.543]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1215320ee05f8ee9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1215320ee05f8ee9"}
21:53:49 [21:53:49.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBZ4iYG/1 with map size 1073741824
21:53:49 [21:53:49.548]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.570]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1215320ee05f8ee9
{"provingJobId":"1:BASE_PARITY:1215320ee05f8ee9"}
21:53:49 [21:53:49.571]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1215320ee05f8ee9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1215320ee05f8ee9"}
21:53:49 [21:53:49.578]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36798a7a62471df2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:36798a7a62471df2"}
21:53:49 [21:53:49.584]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1215320ee05f8ee9 has already been completed
{"provingJobId":"1:BASE_PARITY:1215320ee05f8ee9"}
21:53:49 [21:53:49.590]
INFO:
prover-client:proving-broker Proving Broker started
21:53:49 [21:53:49.591]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a219beda16c1330c epochNumber=1
{"provingJobId":"1:BASE_PARITY:a219beda16c1330c"}
21:53:49 [21:53:49.593]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4xUiqr/1 with map size 1073741824
21:53:49 [21:53:49.598]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:49 [21:53:49.617]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94c06c9cb0f9e67d epochNumber=2
{"provingJobId":"1:BASE_PARITY:94c06c9cb0f9e67d"}
21:53:49 [21:53:49.618]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4xUiqr/2 with map size 1073741824
21:53:49 [21:53:49.621]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:50 [21:53:50.694]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.706]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a219beda16c1330c settled=false
{"provingJobId":"1:BASE_PARITY:a219beda16c1330c","status":"pending"}
21:53:50 [21:53:50.708]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:94c06c9cb0f9e67d settled=false
{"provingJobId":"1:BASE_PARITY:94c06c9cb0f9e67d","status":"pending"}
21:53:50 [21:53:50.709]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a219beda16c1330c type=BASE_PARITY startedAt=1750110829639 older agent has taken job
{"provingJobId":"1:BASE_PARITY:a219beda16c1330c"}
21:53:50 [21:53:50.709]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a219beda16c1330c type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:a219beda16c1330c"}
21:53:50 [21:53:50.713]
INFO:
prover-client:proving-broker Proving Broker started
21:53:50 [21:53:50.714]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf90ce49155842ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf90ce49155842ec"}
21:53:50 [21:53:50.715]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtwc2Fe/1 with map size 1073741824
21:53:50 [21:53:50.717]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:50 [21:53:50.737]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4c9b5f233e9ca0e epochNumber=2
{"provingJobId":"1:BASE_PARITY:d4c9b5f233e9ca0e"}
21:53:50 [21:53:50.737]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtwc2Fe/2 with map size 1073741824
21:53:50 [21:53:50.740]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:51 [21:53:51.757]
INFO:
prover-client:proving-broker Proving Broker started
21:53:51 [21:53:51.758]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bf90ce49155842ec settled=false
{"provingJobId":"1:BASE_PARITY:bf90ce49155842ec","status":"pending"}
21:53:51 [21:53:51.759]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d4c9b5f233e9ca0e settled=false
{"provingJobId":"1:BASE_PARITY:d4c9b5f233e9ca0e","status":"pending"}
21:53:51 [21:53:51.760]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bf90ce49155842ec type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:bf90ce49155842ec"}
21:53:51 [21:53:51.763]
INFO:
prover-client:proving-broker Proving Broker started
21:53:51 [21:53:51.764]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f76fd321062cc6b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f76fd321062cc6b0"}
21:53:51 [21:53:51.765]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvxxoTL/1 with map size 1073741824
21:53:51 [21:53:51.766]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:51 [21:53:51.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ba30d83243504c5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5ba30d83243504c5"}
21:53:51 [21:53:51.786]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvxxoTL/2 with map size 1073741824
21:53:51 [21:53:51.789]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:52 [21:53:52.815]
INFO:
prover-client:proving-broker Proving Broker started
21:53:52 [21:53:52.817]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f76fd321062cc6b0 settled=false
{"provingJobId":"1:BASE_PARITY:f76fd321062cc6b0","status":"pending"}
21:53:52 [21:53:52.820]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5ba30d83243504c5 settled=false
{"provingJobId":"1:BASE_PARITY:5ba30d83243504c5","status":"pending"}
21:53:52 [21:53:52.821]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f76fd321062cc6b0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f76fd321062cc6b0"}
21:53:52 [21:53:52.821]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f76fd321062cc6b0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f76fd321062cc6b0"}
21:53:52 [21:53:52.832]
INFO:
prover-client:proving-broker Proving Broker started
21:53:52 [21:53:52.833]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9c6675ca0f964b7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e9c6675ca0f964b7"}
21:53:52 [21:53:52.833]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBjXnIW/1 with map size 1073741824
21:53:52 [21:53:52.835]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:52 [21:53:52.853]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d5d6c9e94dcb749 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8d5d6c9e94dcb749"}
21:53:52 [21:53:52.855]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testBjXnIW/2 with map size 1073741824
21:53:52 [21:53:52.857]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:52 [21:53:52.875]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e9c6675ca0f964b7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e9c6675ca0f964b7"}
21:53:52 [21:53:52.883]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8d5d6c9e94dcb749 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8d5d6c9e94dcb749"}
21:53:52 [21:53:52.893]
INFO:
prover-client:proving-broker Proving Broker started
21:53:52 [21:53:52.894]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96f412fb5cb3dacc epochNumber=1
{"provingJobId":"1:BASE_PARITY:96f412fb5cb3dacc"}
21:53:52 [21:53:52.895]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrXQMud/1 with map size 1073741824
21:53:52 [21:53:52.897]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:52 [21:53:52.917]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d53b297f1362eb1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2d53b297f1362eb1"}
21:53:52 [21:53:52.918]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrXQMud/2 with map size 1073741824
21:53:52 [21:53:52.921]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:52 [21:53:52.939]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:96f412fb5cb3dacc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:96f412fb5cb3dacc"}
21:53:52 [21:53:52.940]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:96f412fb5cb3dacc type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:96f412fb5cb3dacc"}
21:53:52 [21:53:52.946]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2d53b297f1362eb1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2d53b297f1362eb1"}
21:53:52 [21:53:52.956]
INFO:
prover-client:proving-broker Proving Broker started
21:53:52 [21:53:52.957]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8dcdb8ab308236f5 err=test error
{"provingJoId":"1:BASE_PARITY:8dcdb8ab308236f5"}
21:53:52 [21:53:52.962]
INFO:
prover-client:proving-broker Proving Broker started
21:53:52 [21:53:52.963]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f9e194d0637591bb not found
{"provingJobId":"1:BASE_PARITY:f9e194d0637591bb"}
21:53:52 [21:53:52.966]
INFO:
prover-client:proving-broker Proving Broker started
21:53:52 [21:53:52.967]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5bad47bcf84b3474 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5bad47bcf84b3474"}
21:53:52 [21:53:52.968]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testun5j6e/1 with map size 1073741824
21:53:52 [21:53:52.970]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:52 [21:53:52.988]
INFO:
prover-client:proving-broker Proving Broker started
21:53:52 [21:53:52.988]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1026f8669d123d0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1026f8669d123d0f"}
21:53:52 [21:53:52.989]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testL6qlQP/1 with map size 1073741824
21:53:52 [21:53:52.993]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.139]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1026f8669d123d0f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1026f8669d123d0f"}
21:53:53 [21:53:53.145]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.146]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6647fa28ca5ba341 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6647fa28ca5ba341"}
21:53:53 [21:53:53.148]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFmRoDz/1 with map size 1073741824
21:53:53 [21:53:53.151]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.272]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6647fa28ca5ba341 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6647fa28ca5ba341"}
21:53:53 [21:53:53.274]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9fae14bd666073a8 epochNumber=10
{"provingJobId":"1:BASE_PARITY:9fae14bd666073a8"}
21:53:53 [21:53:53.275]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testFmRoDz/10 with map size 1073741824
21:53:53 [21:53:53.277]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.299]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:53 [21:53:53.300]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:53:53 [21:53:53.428]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.429]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:650d3f6298aea842 epochNumber=1
{"provingJobId":"1:BASE_PARITY:650d3f6298aea842"}
21:53:53 [21:53:53.431]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJaEpKU/1 with map size 1073741824
21:53:53 [21:53:53.434]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.630]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:650d3f6298aea842 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:650d3f6298aea842"}
21:53:53 [21:53:53.663]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.664]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78adfa93dca2e240 epochNumber=1
{"provingJobId":"1:BASE_PARITY:78adfa93dca2e240"}
21:53:53 [21:53:53.665]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testltTR8n/1 with map size 1073741824
21:53:53 [21:53:53.667]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.683]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:78adfa93dca2e240 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:78adfa93dca2e240"}
21:53:53 [21:53:53.686]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.687]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:515001da70c5c4e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:515001da70c5c4e2"}
21:53:53 [21:53:53.688]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWTVBUk/1 with map size 1073741824
21:53:53 [21:53:53.690]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.706]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:515001da70c5c4e2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:515001da70c5c4e2"}
21:53:53 [21:53:53.707]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:515001da70c5c4e2 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:515001da70c5c4e2"}
21:53:53 [21:53:53.716]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd11464f9c38ab46 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd11464f9c38ab46"}
21:53:53 [21:53:53.719]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testF9xGU2/1 with map size 1073741824
21:53:53 [21:53:53.723]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.740]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fd11464f9c38ab46 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fd11464f9c38ab46"}
21:53:53 [21:53:53.749]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.750]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b755b64b398d6c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b755b64b398d6c8"}
21:53:53 [21:53:53.750]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiopKVS/1 with map size 1073741824
21:53:53 [21:53:53.752]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.769]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0b755b64b398d6c8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0b755b64b398d6c8"}
21:53:53 [21:53:53.770]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f0192e226e6ba2f7 epochNumber=3
{"provingJobId":"1:BASE_PARITY:f0192e226e6ba2f7"}
21:53:53 [21:53:53.771]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testiopKVS/3 with map size 1073741824
21:53:53 [21:53:53.773]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.800]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:53 [21:53:53.801]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:53:53 [21:53:53.816]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0b755b64b398d6c8 err=test error
{"provingJoId":"1:BASE_PARITY:0b755b64b398d6c8"}
21:53:53 [21:53:53.821]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBDVsGa/1 with map size 1073741824
21:53:53 [21:53:53.823]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.840]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testBDVsGa/2 with map size 1073741824
21:53:53 [21:53:53.843]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.859]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.862]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b56a88d55b4f92fa settled=false
{"provingJobId":"1:BASE_PARITY:b56a88d55b4f92fa","status":"pending"}
21:53:53 [21:53:53.864]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cf7de32d2e12c6c5 settled=false
{"provingJobId":"1:BASE_PARITY:cf7de32d2e12c6c5","status":"pending"}
21:53:53 [21:53:53.871]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCjQGZX/1 with map size 1073741824
21:53:53 [21:53:53.873]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.890]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCjQGZX/2 with map size 1073741824
21:53:53 [21:53:53.892]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.919]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.922]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a5bc195f257edd66 settled=true
{"provingJobId":"1:BASE_PARITY:a5bc195f257edd66","status":"fulfilled"}
21:53:53 [21:53:53.924]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:a0ad3d0df7456821 settled=true
{"provingJobId":"2:BASE_PARITY:a0ad3d0df7456821","status":"fulfilled"}
21:53:53 [21:53:53.932]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhmL8kR/1 with map size 1073741824
21:53:53 [21:53:53.934]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.958]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testhmL8kR/2 with map size 1073741824
21:53:53 [21:53:53.960]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:53 [21:53:53.979]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.980]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d796c18631ba57a4 settled=true
{"provingJobId":"1:BASE_PARITY:d796c18631ba57a4","status":"fulfilled"}
21:53:53 [21:53:53.983]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:90ef5947369cce86 settled=false
{"provingJobId":"1:BASE_PARITY:90ef5947369cce86","status":"pending"}
21:53:53 [21:53:53.988]
INFO:
prover-client:proving-broker Proving Broker started
21:53:53 [21:53:53.989]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b9cc452c2347567 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7b9cc452c2347567"}
21:53:53 [21:53:53.989]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFuPO84/1 with map size 1073741824
21:53:53 [21:53:53.994]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.023]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.024]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78e0295082ff5cec epochNumber=1
{"provingJobId":"1:BASE_PARITY:78e0295082ff5cec"}
21:53:54 [21:53:54.025]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:78e0295082ff5cec: Error: db error: Error: db error
21:53:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:78e0295082ff5cec"}
21:53:54 [21:53:54.029]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.030]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50832f81b4ea3d9c epochNumber=1
{"provingJobId":"1:BASE_PARITY:50832f81b4ea3d9c"}
21:53:54 [21:53:54.030]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlI1NAN/1 with map size 1073741824
21:53:54 [21:53:54.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.049]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:50832f81b4ea3d9c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:50832f81b4ea3d9c"}
21:53:54 [21:53:54.050]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:50832f81b4ea3d9c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:50832f81b4ea3d9c"}
21:53:54 [21:53:54.060]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.061]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:67fd7adaf66dbb9b epochNumber=1
{"provingJobId":"1:BASE_PARITY:67fd7adaf66dbb9b"}
21:53:54 [21:53:54.062]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfHQVar/1 with map size 1073741824
21:53:54 [21:53:54.064]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.085]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:67fd7adaf66dbb9b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:67fd7adaf66dbb9b"}
21:53:54 [21:53:54.086]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:67fd7adaf66dbb9b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:67fd7adaf66dbb9b"}
21:53:54 [21:53:54.087]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:67fd7adaf66dbb9b: Error: db error
21:53:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:67fd7adaf66dbb9b"}
21:53:54 [21:53:54.093]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.093]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:330c8158440ab05e epochNumber=1
{"provingJobId":"1:BASE_PARITY:330c8158440ab05e"}
21:53:54 [21:53:54.094]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvw8f2J/1 with map size 1073741824
21:53:54 [21:53:54.096]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.112]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:330c8158440ab05e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:330c8158440ab05e"}
21:53:54 [21:53:54.113]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:330c8158440ab05e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:330c8158440ab05e"}
21:53:54 [21:53:54.126]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32b2c3cef2925651 epochNumber=1
{"provingJobId":"1:BASE_PARITY:32b2c3cef2925651"}
21:53:54 [21:53:54.128]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testozRlDW/1 with map size 1073741824
21:53:54 [21:53:54.133]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.152]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:32b2c3cef2925651 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:32b2c3cef2925651"}
21:53:54 [21:53:54.153]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:32b2c3cef2925651 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:32b2c3cef2925651"}
21:53:54 [21:53:54.154]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:32b2c3cef2925651 jobErr=test error: Error: db error
21:53:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:32b2c3cef2925651"}
21:53:54 [21:53:54.161]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.162]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4931474d618b63f5 not found
{"provingJobId":"1:BASE_PARITY:4931474d618b63f5"}
21:53:54 [21:53:54.177]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.178]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:85871e97c778c8f9 err=test error
{"provingJoId":"1:BASE_PARITY:85871e97c778c8f9"}
21:53:54 [21:53:54.183]
INFO:
prover-client:proving-broker Proving Broker started
21:53:54 [21:53:54.195]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b833c2990f02c98e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b833c2990f02c98e"}
21:53:54 [21:53:54.196]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testP90ARI/1 with map size 1073741824
21:53:54 [21:53:54.198]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.221]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b833c2990f02c98e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b833c2990f02c98e"}
21:53:54 [21:53:54.222]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b833c2990f02c98e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b833c2990f02c98e"}
21:53:54 [21:53:54.239]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:704a3900ee4eecc5 epochNumber=2
{"provingJobId":"2:BASE_PARITY:704a3900ee4eecc5"}
21:53:54 [21:53:54.240]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testP90ARI/2 with map size 1073741824
21:53:54 [21:53:54.242]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.261]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:704a3900ee4eecc5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:704a3900ee4eecc5"}
21:53:54 [21:53:54.261]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:704a3900ee4eecc5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:704a3900ee4eecc5"}
21:53:54 [21:53:54.328]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:c2837aad8ea41db0 epochNumber=3
{"provingJobId":"3:BASE_PARITY:c2837aad8ea41db0"}
21:53:54 [21:53:54.329]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testP90ARI/3 with map size 1073741824
21:53:54 [21:53:54.332]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.363]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:54 [21:53:54.363]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:53:54 [21:53:54.413]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:9a13bd9a4bd4c445 epochNumber=4
{"provingJobId":"4:BASE_PARITY:9a13bd9a4bd4c445"}
21:53:54 [21:53:54.414]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testP90ARI/4 with map size 1073741824
21:53:54 [21:53:54.417]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.446]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:54 [21:53:54.446]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
21:53:54 [21:53:54.497]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:53538c51ca9890ec epochNumber=5
{"provingJobId":"5:BASE_PARITY:53538c51ca9890ec"}
21:53:54 [21:53:54.503]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testP90ARI/5 with map size 1073741824
21:53:54 [21:53:54.506]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:53:54 [21:53:54.528]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:53:54 [21:53:54.528]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
21:53:54
PASS src/proving_broker/proving_broker.test.ts (
13.313 s)
21:53:54 ProvingBroker
21:53:54 Producer API
21:53:54
✓ refuses stale jobs (50 ms)
21:53:54
✓ enqueues jobs (3 ms)
21:53:54
✓ ignores duplicate jobs (3 ms)
21:53:54
✓ reports correct status when enqueuing repeat jobs (3 ms)
21:53:54
✓ throws an error in case of duplicate job IDs (20 ms)
21:53:54
✓ returns not-found status for non-existing jobs (2 ms)
21:53:54
✓ cancels jobs in queue (3 ms)
21:53:54
✓ cancels jobs in-progress (2 ms)
21:53:54
✓ returns job result if successful (2 ms)
21:53:54
✓ returns job error if failed (2 ms)
21:53:54
✓ correctly returns job status for concurrent writes (2 ms)
21:53:54
✓ refuses stale jobs (46 ms)
21:53:54
✓ enqueues jobs (43 ms)
21:53:54
✓ ignores duplicate jobs (33 ms)
21:53:54
✓ reports correct status when enqueuing repeat jobs (48 ms)
21:53:54
✓ throws an error in case of duplicate job IDs (25 ms)
21:53:54
✓ returns not-found status for non-existing jobs (3 ms)
21:53:54
✓ cancels jobs in queue (33 ms)
21:53:54
✓ cancels jobs in-progress (48 ms)
21:53:54
✓ returns job result if successful (36 ms)
21:53:54
✓ returns job error if failed (36 ms)
21:53:54
✓ correctly returns job status for concurrent writes (41 ms)
21:53:54 Consumer API
21:53:54
✓ returns undefined if no jobs are available (1 ms)
21:53:54
✓ returns jobs in priority order (2 ms)
21:53:54
✓ returns undefined if no jobs are available for the given allowList (1 ms)
21:53:54
✓ returns a job if it is in the allowList (2 ms)
21:53:54
✓ returns the most important job if it is in the allowList (1 ms)
21:53:54
✓ returns any job if filter is empty (2 ms)
21:53:54
✓ returns a new job when reporting job success (2 ms)
21:53:54
✓ returns a new job when reporting permanent error (1 ms)
21:53:54
✓ returns a new job when reporting retry-able error (2 ms)
21:53:54
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
21:53:54
✓ returns a new job if job is already in progress elsewhere (1072 ms)
21:53:54
✓ avoids sending the same job to a new agent after a restart (1005 ms)
21:53:54
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
21:53:54
✓ tracks job result if in progress (3 ms)
21:53:54
✓ tracks job result even if job is in queue (3 ms)
21:53:54
✓ ignores reported job error if unknown job (2 ms)
21:53:54
✓ ignores job result if unknown job (1 ms)
21:53:54
✓ returns undefined if no jobs are available (2 ms)
21:53:54
✓ returns jobs in priority order (50 ms)
21:53:54
✓ returns undefined if no jobs are available for the given allowList (32 ms)
21:53:54
✓ returns a job if it is in the allowList (64 ms)
21:53:54
✓ returns the most important job if it is in the allowList (60 ms)
21:53:54
✓ returns any job if filter is empty (61 ms)
21:53:54
✓ returns a new job when reporting job success (36 ms)
21:53:54
✓ returns a new job when reporting permanent error (36 ms)
21:53:54
✓ returns a new job when reporting retry-able error (44 ms)
21:53:54
✓ returns a new job when reporting progress if current one is cancelled (50 ms)
21:53:54
✓ returns a new job if job is already in progress elsewhere (1124 ms)
21:53:54
✓ avoids sending the same job to a new agent after a restart (1050 ms)
21:53:54
✓ avoids sending a completed job to a new agent after a restart (1067 ms)
21:53:54
✓ tracks job result if in progress (60 ms)
21:53:54
✓ tracks job result even if job is in queue (62 ms)
21:53:54
✓ ignores reported job error if unknown job (5 ms)
21:53:54
✓ ignores job result if unknown job (5 ms)
21:53:54 Timeouts
21:53:54
✓ tracks in progress jobs (1 ms)
21:53:54
✓ re-enqueues jobs that time out (109 ms)
21:53:54
✓ cancel stale jobs that time out (231 ms)
21:53:54
✓ keeps the jobs in progress while it is alive (203 ms)
21:53:54
✓ tracks in progress jobs (21 ms)
21:53:54
✓ re-enqueues jobs that time out (157 ms)
21:53:54
✓ cancel stale jobs that time out (280 ms)
21:53:54
✓ keeps the jobs in progress while it is alive (237 ms)
21:53:54 Retries
21:53:54
✓ retries jobs (3 ms)
21:53:54
✓ retries up to a maximum number of times (2 ms)
21:53:54
✓ passing retry=false does not retry (2 ms)
21:53:54
✓ does not retry if job is stale (30 ms)
21:53:54
✓ retries jobs (22 ms)
21:53:54
✓ retries up to a maximum number of times (29 ms)
21:53:54
✓ passing retry=false does not retry (33 ms)
21:53:54
✓ does not retry if job is stale (71 ms)
21:53:54 Database management
21:53:54
✓ re-enqueues proof requests on start (3 ms)
21:53:54
✓ restores proof results on start (2 ms)
21:53:54
✓ only re-enqueues unfinished jobs (1 ms)
21:53:54
✓ saves job when enqueued (3 ms)
21:53:54
✓ does not retain job if database fails to save (2 ms)
21:53:54
✓ saves job result (1 ms)
21:53:54
✓ saves result even if database fails to save (2 ms)
21:53:54
✓ saves job error (2 ms)
21:53:54
✓ saves job error even if database fails to save (3 ms)
21:53:54
✓ does not save job result if job is unknown (1 ms)
21:53:54
✓ does not save job error if job is unknown (1 ms)
21:53:54
✓ cleans up old jobs periodically (260 ms)
21:53:54
✓ re-enqueues proof requests on start (49 ms)
21:53:54
✓ restores proof results on start (59 ms)
21:53:54
✓ only re-enqueues unfinished jobs (58 ms)
21:53:54
✓ saves job when enqueued (32 ms)
21:53:54
✓ does not retain job if database fails to save (7 ms)
21:53:54
✓ saves job result (31 ms)
21:53:54
✓ saves result even if database fails to save (31 ms)
21:53:54
✓ saves job error (34 ms)
21:53:54
✓ saves job error even if database fails to save (33 ms)
21:53:54
✓ does not save job result if job is unknown (6 ms)
21:53:54
✓ does not save job error if job is unknown (13 ms)
21:53:54
✓ cleans up old jobs periodically (401 ms)
21:53:54
21:53:54
Test Suites: 1 passed, 1 total
21:53:54
Tests: 96 passed, 96 total
21:53:54
Snapshots: 0 total
21:53:54
Time: 13.444 s
21:53:54
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
21:53:54
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?