Parent Log:
http://ci.aztec-labs.com/af1e20c8fae4f49c
Command: a668859b212fc7b1 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/3c1b9d57eadcc8fb03a7ed42610cf0a1ae4a32eb
Env: REF_NAME=gh-readonly-queue/next/pr-14858-4b0e4dc09fa55f0bc808fa0c7c3e95e8fa997069 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 17 21:40:08 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14858_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
21:40:13 [21:40:13.364]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.372]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:813d9ef30b2e6224 epochNumber=42
{"provingJobId":"1:BASE_PARITY:813d9ef30b2e6224"}
21:40:13 [21:40:13.376]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:3b7ed8fe1c5c2018 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b7ed8fe1c5c2018"}
21:40:13 [21:40:13.415]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.417]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:059f34393eed66e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:059f34393eed66e2"}
21:40:13 [21:40:13.418]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5bca4f676696e0d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5bca4f676696e0d6"}
21:40:13 [21:40:13.420]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.421]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80f6aca21080da96 epochNumber=1
{"provingJobId":"1:BASE_PARITY:80f6aca21080da96"}
21:40:13 [21:40:13.422]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:80f6aca21080da96 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:80f6aca21080da96"}
21:40:13 [21:40:13.424]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.425]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c90349a938479ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c90349a938479ac"}
21:40:13 [21:40:13.426]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8c90349a938479ac epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8c90349a938479ac"}
21:40:13 [21:40:13.427]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8c90349a938479ac type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8c90349a938479ac"}
21:40:13 [21:40:13.427]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8c90349a938479ac epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:8c90349a938479ac"}
21:40:13 [21:40:13.429]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.429]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cb7b522c35f7c3b epochNumber=1
{"provingJobId":"1:BASE_PARITY:0cb7b522c35f7c3b"}
21:40:13 [21:40:13.466]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.469]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.471]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06a5da54812fc91d epochNumber=1
{"provingJobId":"1:BASE_PARITY:06a5da54812fc91d"}
21:40:13 [21:40:13.472]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:06a5da54812fc91d
{"provingJobId":"1:BASE_PARITY:06a5da54812fc91d"}
21:40:13 [21:40:13.473]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:06a5da54812fc91d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:06a5da54812fc91d"}
21:40:13 [21:40:13.473]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:06a5da54812fc91d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:06a5da54812fc91d"}
21:40:13 [21:40:13.476]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.477]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e27f7739d6b902c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e27f7739d6b902c7"}
21:40:13 [21:40:13.479]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e27f7739d6b902c7
{"provingJobId":"1:BASE_PARITY:e27f7739d6b902c7"}
21:40:13 [21:40:13.480]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e27f7739d6b902c7 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:e27f7739d6b902c7"}
21:40:13 [21:40:13.483]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.485]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:418bc6fcdcba6c89 epochNumber=1
{"provingJobId":"1:BASE_PARITY:418bc6fcdcba6c89"}
21:40:13 [21:40:13.486]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:418bc6fcdcba6c89 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:418bc6fcdcba6c89"}
21:40:13 [21:40:13.486]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:418bc6fcdcba6c89 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:418bc6fcdcba6c89"}
21:40:13 [21:40:13.488]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.489]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cbd19312fca1b874 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cbd19312fca1b874"}
21:40:13 [21:40:13.489]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cbd19312fca1b874 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:cbd19312fca1b874"}
21:40:13 [21:40:13.489]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cbd19312fca1b874 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:cbd19312fca1b874"}
21:40:13 [21:40:13.491]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.492]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41856cf6d8280c1e epochNumber=0
{"provingJobId":"1:BASE_PARITY:41856cf6d8280c1e"}
21:40:13 [21:40:13.492]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:41856cf6d8280c1e epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:41856cf6d8280c1e"}
21:40:13 [21:40:13.492]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:18ef156fb96b0d44 epochNumber=0
{"provingJobId":"1:BASE_PARITY:18ef156fb96b0d44"}
21:40:13 [21:40:13.493]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:41856cf6d8280c1e epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:41856cf6d8280c1e"}
21:40:13 [21:40:13.493]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a5dcf3c5434afee epochNumber=0
{"provingJobId":"1:BASE_PARITY:2a5dcf3c5434afee"}
21:40:13 [21:40:13.494]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.496]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.496]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4df194ec72a53e0f epochNumber=2
{"provingJobId":"1:BASE_PARITY:4df194ec72a53e0f"}
21:40:13 [21:40:13.497]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d2035e63b9299ad3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d2035e63b9299ad3"}
21:40:13 [21:40:13.499]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01929ba81f67d2a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01929ba81f67d2a0"}
21:40:13 [21:40:13.501]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.502]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb808d0bb9acd631 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb808d0bb9acd631"}
21:40:13 [21:40:13.503]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4795aa84595b4326 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4795aa84595b4326"}
21:40:13 [21:40:13.503]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35d3e168d9e3d439 epochNumber=2
{"provingJobId":"1:BASE_PARITY:35d3e168d9e3d439"}
21:40:13 [21:40:13.504]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cf43b85a72bd83e epochNumber=1
{"provingJobId":"1:BASE_PARITY:0cf43b85a72bd83e"}
21:40:13 [21:40:13.506]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.509]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c3251ab534e06c50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c3251ab534e06c50"}
21:40:13 [21:40:13.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:638ac97e3edbe441 epochNumber=1
{"provingJobId":"1:BASE_PARITY:638ac97e3edbe441"}
21:40:13 [21:40:13.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fcbffe8101fe87d1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fcbffe8101fe87d1"}
21:40:13 [21:40:13.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9cbef384b3334c5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:9cbef384b3334c5e"}
21:40:13 [21:40:13.512]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:678a6b044acab689 epochNumber=1
{"provingJobId":"1:BASE_PARITY:678a6b044acab689"}
21:40:13 [21:40:13.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3126acb86d91cf71 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3126acb86d91cf71"}
21:40:13 [21:40:13.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f27c8e7153c9308 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0f27c8e7153c9308"}
21:40:13 [21:40:13.514]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8bdd9dcb6afc5147 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8bdd9dcb6afc5147"}
21:40:13 [21:40:13.515]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.516]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b37a8a0cd3f9346 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b37a8a0cd3f9346"}
21:40:13 [21:40:13.517]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c464b2ab6510b82e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c464b2ab6510b82e"}
21:40:13 [21:40:13.517]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3b37a8a0cd3f9346 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3b37a8a0cd3f9346"}
21:40:13 [21:40:13.519]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.519]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0813a170f2e53c9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:0813a170f2e53c9f"}
21:40:13 [21:40:13.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1741c77010b68d5c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1741c77010b68d5c"}
21:40:13 [21:40:13.520]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0813a170f2e53c9f type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:0813a170f2e53c9f"}
21:40:13 [21:40:13.521]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.522]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e140bcbaea76480 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e140bcbaea76480"}
21:40:13 [21:40:13.522]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77553665b587d654 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77553665b587d654"}
21:40:13 [21:40:13.523]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3e140bcbaea76480 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:3e140bcbaea76480"}
21:40:13 [21:40:13.524]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.524]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab6ebfefaaaf4ba1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab6ebfefaaaf4ba1"}
21:40:13 [21:40:13.525]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ab6ebfefaaaf4ba1
{"provingJobId":"1:BASE_PARITY:ab6ebfefaaaf4ba1"}
21:40:13 [21:40:13.525]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ab6ebfefaaaf4ba1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ab6ebfefaaaf4ba1"}
21:40:13 [21:40:13.526]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05ee07aed82ec604 epochNumber=1
{"provingJobId":"1:BASE_PARITY:05ee07aed82ec604"}
21:40:13 [21:40:13.526]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ab6ebfefaaaf4ba1 has already been completed
{"provingJobId":"1:BASE_PARITY:ab6ebfefaaaf4ba1"}
21:40:13 [21:40:13.527]
INFO:
prover-client:proving-broker Proving Broker started
21:40:13 [21:40:13.528]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:064b6e0932e2d955 epochNumber=1
{"provingJobId":"1:BASE_PARITY:064b6e0932e2d955"}
21:40:13 [21:40:13.528]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b64a6d4238c92186 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b64a6d4238c92186"}
21:40:14 [21:40:14.585]
INFO:
prover-client:proving-broker Proving Broker started
21:40:14 [21:40:14.586]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:064b6e0932e2d955 settled=false
{"provingJobId":"1:BASE_PARITY:064b6e0932e2d955","status":"pending"}
21:40:14 [21:40:14.586]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b64a6d4238c92186 settled=false
{"provingJobId":"1:BASE_PARITY:b64a6d4238c92186","status":"pending"}
21:40:14 [21:40:14.587]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:064b6e0932e2d955 type=BASE_PARITY startedAt=1750196413528 older agent has taken job
{"provingJobId":"1:BASE_PARITY:064b6e0932e2d955"}
21:40:14 [21:40:14.587]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:064b6e0932e2d955 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:064b6e0932e2d955"}
21:40:14 [21:40:14.588]
INFO:
prover-client:proving-broker Proving Broker started
21:40:14 [21:40:14.589]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36cd1e3e853ef880 epochNumber=1
{"provingJobId":"1:BASE_PARITY:36cd1e3e853ef880"}
21:40:14 [21:40:14.590]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7913c04ba418a30 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e7913c04ba418a30"}
21:40:15 [21:40:15.591]
INFO:
prover-client:proving-broker Proving Broker started
21:40:15 [21:40:15.592]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:36cd1e3e853ef880 settled=false
{"provingJobId":"1:BASE_PARITY:36cd1e3e853ef880","status":"pending"}
21:40:15 [21:40:15.593]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e7913c04ba418a30 settled=false
{"provingJobId":"1:BASE_PARITY:e7913c04ba418a30","status":"pending"}
21:40:15 [21:40:15.594]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:36cd1e3e853ef880 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:36cd1e3e853ef880"}
21:40:15 [21:40:15.596]
INFO:
prover-client:proving-broker Proving Broker started
21:40:15 [21:40:15.598]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b02457131ee1c02 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b02457131ee1c02"}
21:40:15 [21:40:15.599]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b6d5d93cd415462 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4b6d5d93cd415462"}
21:40:16 [21:40:16.599]
INFO:
prover-client:proving-broker Proving Broker started
21:40:16 [21:40:16.600]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3b02457131ee1c02 settled=false
{"provingJobId":"1:BASE_PARITY:3b02457131ee1c02","status":"pending"}
21:40:16 [21:40:16.600]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4b6d5d93cd415462 settled=false
{"provingJobId":"1:BASE_PARITY:4b6d5d93cd415462","status":"pending"}
21:40:16 [21:40:16.600]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3b02457131ee1c02 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3b02457131ee1c02"}
21:40:16 [21:40:16.601]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3b02457131ee1c02 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3b02457131ee1c02"}
21:40:16 [21:40:16.602]
INFO:
prover-client:proving-broker Proving Broker started
21:40:16 [21:40:16.602]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a359dadb49b6681 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a359dadb49b6681"}
21:40:16 [21:40:16.602]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97fdf653535d6a47 epochNumber=2
{"provingJobId":"1:BASE_PARITY:97fdf653535d6a47"}
21:40:16 [21:40:16.604]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5a359dadb49b6681 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5a359dadb49b6681"}
21:40:16 [21:40:16.605]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:97fdf653535d6a47 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:97fdf653535d6a47"}
21:40:16 [21:40:16.606]
INFO:
prover-client:proving-broker Proving Broker started
21:40:16 [21:40:16.607]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d497aac1c83cb302 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d497aac1c83cb302"}
21:40:16 [21:40:16.607]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e8733e773e0b8ef epochNumber=2
{"provingJobId":"1:BASE_PARITY:8e8733e773e0b8ef"}
21:40:16 [21:40:16.608]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d497aac1c83cb302 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d497aac1c83cb302"}
21:40:16 [21:40:16.608]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d497aac1c83cb302 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d497aac1c83cb302"}
21:40:16 [21:40:16.608]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8e8733e773e0b8ef type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8e8733e773e0b8ef"}
21:40:16 [21:40:16.610]
INFO:
prover-client:proving-broker Proving Broker started
21:40:16 [21:40:16.610]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bc7627dcf5a45829 err=test error
{"provingJoId":"1:BASE_PARITY:bc7627dcf5a45829"}
21:40:16 [21:40:16.611]
INFO:
prover-client:proving-broker Proving Broker started
21:40:16 [21:40:16.612]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0d95cef3f10846d3 not found
{"provingJobId":"1:BASE_PARITY:0d95cef3f10846d3"}
21:40:16 [21:40:16.614]
INFO:
prover-client:proving-broker Proving Broker started
21:40:16 [21:40:16.614]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:531fb1bb2890c45b epochNumber=1
{"provingJobId":"1:BASE_PARITY:531fb1bb2890c45b"}
21:40:16 [21:40:16.615]
INFO:
prover-client:proving-broker Proving Broker started
21:40:16 [21:40:16.616]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:203562e13d4193b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:203562e13d4193b6"}
21:40:16 [21:40:16.719]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:203562e13d4193b6 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:203562e13d4193b6"}
21:40:16 [21:40:16.724]
INFO:
prover-client:proving-broker Proving Broker started
21:40:16 [21:40:16.725]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b20982b0bbc9c8d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b20982b0bbc9c8d7"}
21:40:16 [21:40:16.852]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b20982b0bbc9c8d7 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b20982b0bbc9c8d7"}
21:40:16 [21:40:16.855]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0087995f0817c233 epochNumber=10
{"provingJobId":"1:BASE_PARITY:0087995f0817c233"}
21:40:16 [21:40:16.879]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:16 [21:40:16.982]
INFO:
prover-client:proving-broker Proving Broker started
21:40:16 [21:40:16.984]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b969d130704d2f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b969d130704d2f0"}
21:40:17 [21:40:17.135]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3b969d130704d2f0 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:3b969d130704d2f0"}
21:40:17 [21:40:17.188]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.189]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:208af26c7c913711 epochNumber=1
{"provingJobId":"1:BASE_PARITY:208af26c7c913711"}
21:40:17 [21:40:17.190]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:208af26c7c913711 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:208af26c7c913711"}
21:40:17 [21:40:17.192]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.192]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec90dc48d5b9c7a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec90dc48d5b9c7a6"}
21:40:17 [21:40:17.193]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ec90dc48d5b9c7a6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:ec90dc48d5b9c7a6"}
21:40:17 [21:40:17.194]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ec90dc48d5b9c7a6 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:ec90dc48d5b9c7a6"}
21:40:17 [21:40:17.195]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.195]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7bfcac7c0cc42055 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7bfcac7c0cc42055"}
21:40:17 [21:40:17.196]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7bfcac7c0cc42055 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7bfcac7c0cc42055"}
21:40:17 [21:40:17.197]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.197]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa7cecd3ad17a1c5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa7cecd3ad17a1c5"}
21:40:17 [21:40:17.198]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:aa7cecd3ad17a1c5 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:aa7cecd3ad17a1c5"}
21:40:17 [21:40:17.198]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:203c08dd79582931 epochNumber=3
{"provingJobId":"1:BASE_PARITY:203c08dd79582931"}
21:40:17 [21:40:17.222]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:17 [21:40:17.224]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:aa7cecd3ad17a1c5 err=test error
{"provingJoId":"1:BASE_PARITY:aa7cecd3ad17a1c5"}
21:40:17 [21:40:17.226]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.226]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f9bbe015470e35b1 settled=false
{"provingJobId":"1:BASE_PARITY:f9bbe015470e35b1","status":"pending"}
21:40:17 [21:40:17.226]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fcd174303fd07bfb settled=false
{"provingJobId":"1:BASE_PARITY:fcd174303fd07bfb","status":"pending"}
21:40:17 [21:40:17.228]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.229]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2a25c68bcd76723f settled=true
{"provingJobId":"1:BASE_PARITY:2a25c68bcd76723f","status":"fulfilled"}
21:40:17 [21:40:17.229]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:d4bcb5ba0ba58b4a settled=true
{"provingJobId":"2:BASE_PARITY:d4bcb5ba0ba58b4a","status":"fulfilled"}
21:40:17 [21:40:17.230]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.230]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:380d5187c0f79824 settled=true
{"provingJobId":"1:BASE_PARITY:380d5187c0f79824","status":"fulfilled"}
21:40:17 [21:40:17.230]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b1a34adb6974b625 settled=false
{"provingJobId":"1:BASE_PARITY:b1a34adb6974b625","status":"pending"}
21:40:17 [21:40:17.231]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04b3bea79919c464 epochNumber=1
{"provingJobId":"1:BASE_PARITY:04b3bea79919c464"}
21:40:17 [21:40:17.234]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.234]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c01f316ff4abe1dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:c01f316ff4abe1dc"}
21:40:17 [21:40:17.235]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:c01f316ff4abe1dc: Error: db error: Error: db error
21:40:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:c01f316ff4abe1dc"}
21:40:17 [21:40:17.236]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e13f0351b6226e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2e13f0351b6226e8"}
21:40:17 [21:40:17.237]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2e13f0351b6226e8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2e13f0351b6226e8"}
21:40:17 [21:40:17.237]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2e13f0351b6226e8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2e13f0351b6226e8"}
21:40:17 [21:40:17.238]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a0506e417478b5d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a0506e417478b5d4"}
21:40:17 [21:40:17.238]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a0506e417478b5d4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a0506e417478b5d4"}
21:40:17 [21:40:17.239]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a0506e417478b5d4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a0506e417478b5d4"}
21:40:17 [21:40:17.239]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:a0506e417478b5d4: Error: db error
21:40:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:a0506e417478b5d4"}
21:40:17 [21:40:17.240]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.240]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf16dfa757102184 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf16dfa757102184"}
21:40:17 [21:40:17.241]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cf16dfa757102184 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:cf16dfa757102184"}
21:40:17 [21:40:17.241]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cf16dfa757102184 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:cf16dfa757102184"}
21:40:17 [21:40:17.242]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.242]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ece64f19efeeb2b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ece64f19efeeb2b4"}
21:40:17 [21:40:17.242]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ece64f19efeeb2b4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ece64f19efeeb2b4"}
21:40:17 [21:40:17.243]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ece64f19efeeb2b4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ece64f19efeeb2b4"}
21:40:17 [21:40:17.243]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:ece64f19efeeb2b4 jobErr=test error: Error: db error
21:40:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:ece64f19efeeb2b4"}
21:40:17 [21:40:17.244]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.245]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7240065edf88deec not found
{"provingJobId":"1:BASE_PARITY:7240065edf88deec"}
21:40:17 [21:40:17.246]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.246]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ce9cda0e031c5212 err=test error
{"provingJoId":"1:BASE_PARITY:ce9cda0e031c5212"}
21:40:17 [21:40:17.247]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.258]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c6e56abc672eefa epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c6e56abc672eefa"}
21:40:17 [21:40:17.259]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2c6e56abc672eefa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2c6e56abc672eefa"}
21:40:17 [21:40:17.259]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2c6e56abc672eefa type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2c6e56abc672eefa"}
21:40:17 [21:40:17.270]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:c9e10c07c6176167 epochNumber=2
{"provingJobId":"2:BASE_PARITY:c9e10c07c6176167"}
21:40:17 [21:40:17.270]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:c9e10c07c6176167 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:c9e10c07c6176167"}
21:40:17 [21:40:17.270]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:c9e10c07c6176167 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:c9e10c07c6176167"}
21:40:17 [21:40:17.332]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:d3e58cf8e2c62854 epochNumber=3
{"provingJobId":"3:BASE_PARITY:d3e58cf8e2c62854"}
21:40:17 [21:40:17.348]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:17 [21:40:17.395]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:244765d9930392ba epochNumber=4
{"provingJobId":"4:BASE_PARITY:244765d9930392ba"}
21:40:17 [21:40:17.401]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:17 [21:40:17.458]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:15249adf70a21bb1 epochNumber=5
{"provingJobId":"5:BASE_PARITY:15249adf70a21bb1"}
21:40:17 [21:40:17.479]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:17 [21:40:17.513]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.513]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce454dd44fe14bdd epochNumber=42
{"provingJobId":"1:BASE_PARITY:ce454dd44fe14bdd"}
21:40:17 [21:40:17.514]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testfKwGOI/42 with map size 1073741824
21:40:17 [21:40:17.517]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.546]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:269b3d5e133d7775 epochNumber=1
{"provingJobId":"1:BASE_PARITY:269b3d5e133d7775"}
21:40:17 [21:40:17.551]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.552]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89203ab4c93443ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:89203ab4c93443ec"}
21:40:17 [21:40:17.553]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3fqZaN/1 with map size 1073741824
21:40:17 [21:40:17.559]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.584]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ffa146caff99fd67 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ffa146caff99fd67"}
21:40:17 [21:40:17.603]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.604]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ff99284caf5e209 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ff99284caf5e209"}
21:40:17 [21:40:17.606]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtaaInS/1 with map size 1073741824
21:40:17 [21:40:17.609]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.630]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9ff99284caf5e209 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9ff99284caf5e209"}
21:40:17 [21:40:17.634]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.635]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:995c67b1058ff4cc epochNumber=1
{"provingJobId":"1:BASE_PARITY:995c67b1058ff4cc"}
21:40:17 [21:40:17.635]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaQLpQQ/1 with map size 1073741824
21:40:17 [21:40:17.637]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.656]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:995c67b1058ff4cc epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:995c67b1058ff4cc"}
21:40:17 [21:40:17.657]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:995c67b1058ff4cc type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:995c67b1058ff4cc"}
21:40:17 [21:40:17.667]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:995c67b1058ff4cc epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:995c67b1058ff4cc"}
21:40:17 [21:40:17.670]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.671]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f17df37bfbe06f6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f17df37bfbe06f6"}
21:40:17 [21:40:17.673]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testluv63I/1 with map size 1073741824
21:40:17 [21:40:17.676]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.713]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.717]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.718]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1487f4b4ea9a3a45 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1487f4b4ea9a3a45"}
21:40:17 [21:40:17.719]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0Q9g7y/1 with map size 1073741824
21:40:17 [21:40:17.720]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.737]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1487f4b4ea9a3a45
{"provingJobId":"1:BASE_PARITY:1487f4b4ea9a3a45"}
21:40:17 [21:40:17.738]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1487f4b4ea9a3a45 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1487f4b4ea9a3a45"}
21:40:17 [21:40:17.738]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1487f4b4ea9a3a45 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1487f4b4ea9a3a45"}
21:40:17 [21:40:17.750]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.751]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13c10725906b8c24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13c10725906b8c24"}
21:40:17 [21:40:17.752]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test15zlXd/1 with map size 1073741824
21:40:17 [21:40:17.754]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.773]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:13c10725906b8c24
{"provingJobId":"1:BASE_PARITY:13c10725906b8c24"}
21:40:17 [21:40:17.773]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:13c10725906b8c24 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:13c10725906b8c24"}
21:40:17 [21:40:17.798]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7adab04910d3a24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7adab04910d3a24"}
21:40:17 [21:40:17.800]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testshhx1f/1 with map size 1073741824
21:40:17 [21:40:17.802]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.820]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f7adab04910d3a24 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f7adab04910d3a24"}
21:40:17 [21:40:17.820]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f7adab04910d3a24 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f7adab04910d3a24"}
21:40:17 [21:40:17.830]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.831]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8dd5d9cbf26fd582 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8dd5d9cbf26fd582"}
21:40:17 [21:40:17.832]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZJn0xa/1 with map size 1073741824
21:40:17 [21:40:17.835]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.853]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8dd5d9cbf26fd582 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8dd5d9cbf26fd582"}
21:40:17 [21:40:17.853]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8dd5d9cbf26fd582 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8dd5d9cbf26fd582"}
21:40:17 [21:40:17.863]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.864]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:65452fa29594b883 epochNumber=0
{"provingJobId":"1:BASE_PARITY:65452fa29594b883"}
21:40:17 [21:40:17.864]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testat4clD/0 with map size 1073741824
21:40:17 [21:40:17.866]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.892]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:65452fa29594b883 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:65452fa29594b883"}
21:40:17 [21:40:17.897]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d17450dae9944de5 epochNumber=0
{"provingJobId":"1:BASE_PARITY:d17450dae9944de5"}
21:40:17 [21:40:17.898]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:65452fa29594b883 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:65452fa29594b883"}
21:40:17 [21:40:17.898]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cdcb5203b2d799e0 epochNumber=0
{"provingJobId":"1:BASE_PARITY:cdcb5203b2d799e0"}
21:40:17 [21:40:17.912]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.914]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.914]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96ca944ccf730343 epochNumber=2
{"provingJobId":"1:BASE_PARITY:96ca944ccf730343"}
21:40:17 [21:40:17.915]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOUG2te/2 with map size 1073741824
21:40:17 [21:40:17.916]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.931]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8cee0437aa8d8f9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:8cee0437aa8d8f9f"}
21:40:17 [21:40:17.931]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOUG2te/1 with map size 1073741824
21:40:17 [21:40:17.934]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:17.954]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:17.955]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d670059e648c8ca5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d670059e648c8ca5"}
21:40:17 [21:40:17.956]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBDnzcy/1 with map size 1073741824
21:40:17 [21:40:17.959]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:17 [21:40:18.000]
INFO:
prover-client:proving-broker Proving Broker started
21:40:17 [21:40:18.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5934fe7208e47c12 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5934fe7208e47c12"}
21:40:18 [21:40:18.001]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjX3iVH/1 with map size 1073741824
21:40:18 [21:40:18.005]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.024]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92919633f28c2080 epochNumber=1
{"provingJobId":"1:BASE_PARITY:92919633f28c2080"}
21:40:18 [21:40:18.030]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff4b345c3b295046 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ff4b345c3b295046"}
21:40:18 [21:40:18.030]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjX3iVH/2 with map size 1073741824
21:40:18 [21:40:18.033]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.052]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a6ce30ec53679ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a6ce30ec53679ac"}
21:40:18 [21:40:18.062]
INFO:
prover-client:proving-broker Proving Broker started
21:40:18 [21:40:18.062]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77c313e8ffc0dc22 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77c313e8ffc0dc22"}
21:40:18 [21:40:18.063]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXCm8K4/1 with map size 1073741824
21:40:18 [21:40:18.064]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.088]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00b24a268a1e5d57 epochNumber=1
{"provingJobId":"1:BASE_PARITY:00b24a268a1e5d57"}
21:40:18 [21:40:18.097]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eae1cb2108b01f56 epochNumber=2
{"provingJobId":"1:BASE_PARITY:eae1cb2108b01f56"}
21:40:18 [21:40:18.099]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testXCm8K4/2 with map size 1073741824
21:40:18 [21:40:18.102]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.120]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c9006071ba6b550 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c9006071ba6b550"}
21:40:18 [21:40:18.130]
INFO:
prover-client:proving-broker Proving Broker started
21:40:18 [21:40:18.131]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84429b4305a7b475 epochNumber=1
{"provingJobId":"1:BASE_PARITY:84429b4305a7b475"}
21:40:18 [21:40:18.133]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGaDNDS/1 with map size 1073741824
21:40:18 [21:40:18.135]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.152]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:73377e9e2a5a0382 epochNumber=1
{"provingJobId":"1:BASE_PARITY:73377e9e2a5a0382"}
21:40:18 [21:40:18.160]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c29cd3823d6325c2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c29cd3823d6325c2"}
21:40:18 [21:40:18.161]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGaDNDS/2 with map size 1073741824
21:40:18 [21:40:18.165]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.186]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a4a10067037fb23d epochNumber=1
{"provingJobId":"1:BASE_PARITY:a4a10067037fb23d"}
21:40:18 [21:40:18.199]
INFO:
prover-client:proving-broker Proving Broker started
21:40:18 [21:40:18.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84b55b1a98d8f02a epochNumber=1
{"provingJobId":"1:BASE_PARITY:84b55b1a98d8f02a"}
21:40:18 [21:40:18.200]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLlyKMS/1 with map size 1073741824
21:40:18 [21:40:18.202]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.220]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ddc208110900f42d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ddc208110900f42d"}
21:40:18 [21:40:18.227]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:84b55b1a98d8f02a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:84b55b1a98d8f02a"}
21:40:18 [21:40:18.237]
INFO:
prover-client:proving-broker Proving Broker started
21:40:18 [21:40:18.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:69c101bd831e5c9c epochNumber=1
{"provingJobId":"1:BASE_PARITY:69c101bd831e5c9c"}
21:40:18 [21:40:18.239]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbtiT9y/1 with map size 1073741824
21:40:18 [21:40:18.242]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.261]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5cf6d6b14c914816 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5cf6d6b14c914816"}
21:40:18 [21:40:18.268]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:69c101bd831e5c9c type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:69c101bd831e5c9c"}
21:40:18 [21:40:18.288]
INFO:
prover-client:proving-broker Proving Broker started
21:40:18 [21:40:18.290]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7b22b75cc3c0dd6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7b22b75cc3c0dd6"}
21:40:18 [21:40:18.291]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testp3lDgw/1 with map size 1073741824
21:40:18 [21:40:18.298]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.320]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7da88766b9eb113c epochNumber=1
{"provingJobId":"1:BASE_PARITY:7da88766b9eb113c"}
21:40:18 [21:40:18.326]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e7b22b75cc3c0dd6 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:e7b22b75cc3c0dd6"}
21:40:18 [21:40:18.332]
INFO:
prover-client:proving-broker Proving Broker started
21:40:18 [21:40:18.333]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86d115781cca3010 epochNumber=1
{"provingJobId":"1:BASE_PARITY:86d115781cca3010"}
21:40:18 [21:40:18.334]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHyijSA/1 with map size 1073741824
21:40:18 [21:40:18.337]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.355]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:86d115781cca3010
{"provingJobId":"1:BASE_PARITY:86d115781cca3010"}
21:40:18 [21:40:18.356]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:86d115781cca3010 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:86d115781cca3010"}
21:40:18 [21:40:18.362]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90bfde31e44da15a epochNumber=1
{"provingJobId":"1:BASE_PARITY:90bfde31e44da15a"}
21:40:18 [21:40:18.368]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:86d115781cca3010 has already been completed
{"provingJobId":"1:BASE_PARITY:86d115781cca3010"}
21:40:18 [21:40:18.372]
INFO:
prover-client:proving-broker Proving Broker started
21:40:18 [21:40:18.373]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f68994afb98c8b98 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f68994afb98c8b98"}
21:40:18 [21:40:18.373]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6yBbv4/1 with map size 1073741824
21:40:18 [21:40:18.380]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:18 [21:40:18.400]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28beb2c0b486474c epochNumber=2
{"provingJobId":"1:BASE_PARITY:28beb2c0b486474c"}
21:40:18 [21:40:18.401]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6yBbv4/2 with map size 1073741824
21:40:18 [21:40:18.403]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:19 [21:40:19.469]
INFO:
prover-client:proving-broker Proving Broker started
21:40:19 [21:40:19.477]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f68994afb98c8b98 settled=false
{"provingJobId":"1:BASE_PARITY:f68994afb98c8b98","status":"pending"}
21:40:19 [21:40:19.479]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:28beb2c0b486474c settled=false
{"provingJobId":"1:BASE_PARITY:28beb2c0b486474c","status":"pending"}
21:40:19 [21:40:19.480]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f68994afb98c8b98 type=BASE_PARITY startedAt=1750196418418 older agent has taken job
{"provingJobId":"1:BASE_PARITY:f68994afb98c8b98"}
21:40:19 [21:40:19.481]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f68994afb98c8b98 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:f68994afb98c8b98"}
21:40:19 [21:40:19.486]
INFO:
prover-client:proving-broker Proving Broker started
21:40:19 [21:40:19.487]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aaba42130eec27e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aaba42130eec27e7"}
21:40:19 [21:40:19.488]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testobMiK0/1 with map size 1073741824
21:40:19 [21:40:19.490]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:19 [21:40:19.507]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e6e048b6258286d epochNumber=2
{"provingJobId":"1:BASE_PARITY:8e6e048b6258286d"}
21:40:19 [21:40:19.507]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testobMiK0/2 with map size 1073741824
21:40:19 [21:40:19.511]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:20 [21:40:20.528]
INFO:
prover-client:proving-broker Proving Broker started
21:40:20 [21:40:20.529]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:aaba42130eec27e7 settled=false
{"provingJobId":"1:BASE_PARITY:aaba42130eec27e7","status":"pending"}
21:40:20 [21:40:20.530]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8e6e048b6258286d settled=false
{"provingJobId":"1:BASE_PARITY:8e6e048b6258286d","status":"pending"}
21:40:20 [21:40:20.531]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aaba42130eec27e7 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:aaba42130eec27e7"}
21:40:20 [21:40:20.535]
INFO:
prover-client:proving-broker Proving Broker started
21:40:20 [21:40:20.536]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1b519856b0ef33a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1b519856b0ef33a"}
21:40:20 [21:40:20.538]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYVzLge/1 with map size 1073741824
21:40:20 [21:40:20.540]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:20 [21:40:20.559]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb9d6a8faf9f3ef3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cb9d6a8faf9f3ef3"}
21:40:20 [21:40:20.560]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYVzLge/2 with map size 1073741824
21:40:20 [21:40:20.562]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:21 [21:40:21.590]
INFO:
prover-client:proving-broker Proving Broker started
21:40:21 [21:40:21.595]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a1b519856b0ef33a settled=false
{"provingJobId":"1:BASE_PARITY:a1b519856b0ef33a","status":"pending"}
21:40:21 [21:40:21.599]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cb9d6a8faf9f3ef3 settled=false
{"provingJobId":"1:BASE_PARITY:cb9d6a8faf9f3ef3","status":"pending"}
21:40:21 [21:40:21.601]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a1b519856b0ef33a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a1b519856b0ef33a"}
21:40:21 [21:40:21.601]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a1b519856b0ef33a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a1b519856b0ef33a"}
21:40:21 [21:40:21.615]
INFO:
prover-client:proving-broker Proving Broker started
21:40:21 [21:40:21.616]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8cafa8dfadb45e65 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8cafa8dfadb45e65"}
21:40:21 [21:40:21.617]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSdDjfS/1 with map size 1073741824
21:40:21 [21:40:21.619]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:21 [21:40:21.640]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f1f0e960598989a epochNumber=2
{"provingJobId":"1:BASE_PARITY:5f1f0e960598989a"}
21:40:21 [21:40:21.641]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSdDjfS/2 with map size 1073741824
21:40:21 [21:40:21.644]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:21 [21:40:21.664]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8cafa8dfadb45e65 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8cafa8dfadb45e65"}
21:40:21 [21:40:21.673]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5f1f0e960598989a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5f1f0e960598989a"}
21:40:21 [21:40:21.686]
INFO:
prover-client:proving-broker Proving Broker started
21:40:21 [21:40:21.687]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:abca67d869c1380b epochNumber=1
{"provingJobId":"1:BASE_PARITY:abca67d869c1380b"}
21:40:21 [21:40:21.691]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQz7wsa/1 with map size 1073741824
21:40:21 [21:40:21.694]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:21 [21:40:21.713]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:43a3febd520cde54 epochNumber=2
{"provingJobId":"1:BASE_PARITY:43a3febd520cde54"}
21:40:21 [21:40:21.715]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQz7wsa/2 with map size 1073741824
21:40:21 [21:40:21.717]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:21 [21:40:21.733]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:abca67d869c1380b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:abca67d869c1380b"}
21:40:21 [21:40:21.733]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:abca67d869c1380b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:abca67d869c1380b"}
21:40:21 [21:40:21.739]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:43a3febd520cde54 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:43a3febd520cde54"}
21:40:21 [21:40:21.751]
INFO:
prover-client:proving-broker Proving Broker started
21:40:21 [21:40:21.753]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4704f18ccb845961 err=test error
{"provingJoId":"1:BASE_PARITY:4704f18ccb845961"}
21:40:21 [21:40:21.757]
INFO:
prover-client:proving-broker Proving Broker started
21:40:21 [21:40:21.757]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fbac1e2d894af5cd not found
{"provingJobId":"1:BASE_PARITY:fbac1e2d894af5cd"}
21:40:21 [21:40:21.760]
INFO:
prover-client:proving-broker Proving Broker started
21:40:21 [21:40:21.761]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54f3d96b369f828a epochNumber=1
{"provingJobId":"1:BASE_PARITY:54f3d96b369f828a"}
21:40:21 [21:40:21.761]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeMdvJI/1 with map size 1073741824
21:40:21 [21:40:21.763]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:21 [21:40:21.785]
INFO:
prover-client:proving-broker Proving Broker started
21:40:21 [21:40:21.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b91d7bee93e58bb9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b91d7bee93e58bb9"}
21:40:21 [21:40:21.787]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0otoqw/1 with map size 1073741824
21:40:21 [21:40:21.790]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:21 [21:40:21.911]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b91d7bee93e58bb9 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b91d7bee93e58bb9"}
21:40:21 [21:40:21.918]
INFO:
prover-client:proving-broker Proving Broker started
21:40:21 [21:40:21.919]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32c6812906fc2e9c epochNumber=1
{"provingJobId":"1:BASE_PARITY:32c6812906fc2e9c"}
21:40:21 [21:40:21.919]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIPvAp6/1 with map size 1073741824
21:40:21 [21:40:21.921]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.045]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:32c6812906fc2e9c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:32c6812906fc2e9c"}
21:40:22 [21:40:22.052]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16341239ca5b5b12 epochNumber=10
{"provingJobId":"1:BASE_PARITY:16341239ca5b5b12"}
21:40:22 [21:40:22.055]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testIPvAp6/10 with map size 1073741824
21:40:22 [21:40:22.056]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.096]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:22 [21:40:22.097]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:40:22 [21:40:22.209]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.210]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:49200d561da69ef5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:49200d561da69ef5"}
21:40:22 [21:40:22.211]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6plB9L/1 with map size 1073741824
21:40:22 [21:40:22.214]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.387]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:49200d561da69ef5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:49200d561da69ef5"}
21:40:22 [21:40:22.437]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.438]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b1f74a5cd91d3d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b1f74a5cd91d3d0"}
21:40:22 [21:40:22.439]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGWaBAe/1 with map size 1073741824
21:40:22 [21:40:22.444]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.462]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3b1f74a5cd91d3d0 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3b1f74a5cd91d3d0"}
21:40:22 [21:40:22.467]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.468]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f73e87b6a601c86b epochNumber=1
{"provingJobId":"1:BASE_PARITY:f73e87b6a601c86b"}
21:40:22 [21:40:22.469]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMMa14K/1 with map size 1073741824
21:40:22 [21:40:22.472]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.491]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f73e87b6a601c86b type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f73e87b6a601c86b"}
21:40:22 [21:40:22.492]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f73e87b6a601c86b type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:f73e87b6a601c86b"}
21:40:22 [21:40:22.501]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.502]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8021a8c486f3cded epochNumber=1
{"provingJobId":"1:BASE_PARITY:8021a8c486f3cded"}
21:40:22 [21:40:22.505]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDqYKp2/1 with map size 1073741824
21:40:22 [21:40:22.508]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.526]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8021a8c486f3cded type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8021a8c486f3cded"}
21:40:22 [21:40:22.536]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4dfaf1020fc6a15d epochNumber=1
{"provingJobId":"1:BASE_PARITY:4dfaf1020fc6a15d"}
21:40:22 [21:40:22.537]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testy8AUR8/1 with map size 1073741824
21:40:22 [21:40:22.539]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.562]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4dfaf1020fc6a15d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:4dfaf1020fc6a15d"}
21:40:22 [21:40:22.564]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c594d47cc1c16aa epochNumber=3
{"provingJobId":"1:BASE_PARITY:5c594d47cc1c16aa"}
21:40:22 [21:40:22.564]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testy8AUR8/3 with map size 1073741824
21:40:22 [21:40:22.568]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.590]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:22 [21:40:22.591]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:40:22 [21:40:22.610]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4dfaf1020fc6a15d err=test error
{"provingJoId":"1:BASE_PARITY:4dfaf1020fc6a15d"}
21:40:22 [21:40:22.615]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMGqtmt/1 with map size 1073741824
21:40:22 [21:40:22.617]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.635]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMGqtmt/2 with map size 1073741824
21:40:22 [21:40:22.637]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.655]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.658]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:60235abec5595410 settled=false
{"provingJobId":"1:BASE_PARITY:60235abec5595410","status":"pending"}
21:40:22 [21:40:22.659]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0a320359fe178fd9 settled=false
{"provingJobId":"1:BASE_PARITY:0a320359fe178fd9","status":"pending"}
21:40:22 [21:40:22.666]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testivi0fj/1 with map size 1073741824
21:40:22 [21:40:22.669]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.688]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testivi0fj/2 with map size 1073741824
21:40:22 [21:40:22.690]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.725]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.729]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2ff8cf439be87fdd settled=true
{"provingJobId":"1:BASE_PARITY:2ff8cf439be87fdd","status":"fulfilled"}
21:40:22 [21:40:22.731]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:4882b46846afee3f settled=true
{"provingJobId":"2:BASE_PARITY:4882b46846afee3f","status":"fulfilled"}
21:40:22 [21:40:22.739]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIE1jmB/1 with map size 1073741824
21:40:22 [21:40:22.741]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.771]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIE1jmB/2 with map size 1073741824
21:40:22 [21:40:22.775]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.795]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.798]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:47cc273c87d2d6bf settled=true
{"provingJobId":"1:BASE_PARITY:47cc273c87d2d6bf","status":"fulfilled"}
21:40:22 [21:40:22.800]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bb514a2068852f68 settled=false
{"provingJobId":"1:BASE_PARITY:bb514a2068852f68","status":"pending"}
21:40:22 [21:40:22.806]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c7240bd8fe94dbc epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c7240bd8fe94dbc"}
21:40:22 [21:40:22.808]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfrg13R/1 with map size 1073741824
21:40:22 [21:40:22.812]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.839]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.840]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a2172f2d60461b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a2172f2d60461b1"}
21:40:22 [21:40:22.841]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:2a2172f2d60461b1: Error: db error: Error: db error
21:40:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:2a2172f2d60461b1"}
21:40:22 [21:40:22.845]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.845]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7bc3c130f2d125c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7bc3c130f2d125c7"}
21:40:22 [21:40:22.846]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiqkZCH/1 with map size 1073741824
21:40:22 [21:40:22.849]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.870]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7bc3c130f2d125c7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7bc3c130f2d125c7"}
21:40:22 [21:40:22.871]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7bc3c130f2d125c7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7bc3c130f2d125c7"}
21:40:22 [21:40:22.882]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cea058674f04036b epochNumber=1
{"provingJobId":"1:BASE_PARITY:cea058674f04036b"}
21:40:22 [21:40:22.884]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3aCcXl/1 with map size 1073741824
21:40:22 [21:40:22.888]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.906]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cea058674f04036b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:cea058674f04036b"}
21:40:22 [21:40:22.906]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cea058674f04036b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cea058674f04036b"}
21:40:22 [21:40:22.907]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:cea058674f04036b: Error: db error
21:40:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:cea058674f04036b"}
21:40:22 [21:40:22.912]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.912]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f6de1443289a64f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f6de1443289a64f0"}
21:40:22 [21:40:22.913]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMmhEaV/1 with map size 1073741824
21:40:22 [21:40:22.916]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.937]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f6de1443289a64f0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f6de1443289a64f0"}
21:40:22 [21:40:22.937]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f6de1443289a64f0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f6de1443289a64f0"}
21:40:22 [21:40:22.948]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.949]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aaaf6cadc96f6333 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aaaf6cadc96f6333"}
21:40:22 [21:40:22.950]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5aVU3v/1 with map size 1073741824
21:40:22 [21:40:22.954]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:22 [21:40:22.981]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aaaf6cadc96f6333 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:aaaf6cadc96f6333"}
21:40:22 [21:40:22.981]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:aaaf6cadc96f6333 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:aaaf6cadc96f6333"}
21:40:22 [21:40:22.982]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:aaaf6cadc96f6333 jobErr=test error: Error: db error
21:40:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:aaaf6cadc96f6333"}
21:40:22 [21:40:22.988]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.989]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6a2c512bc41a3fd4 not found
{"provingJobId":"1:BASE_PARITY:6a2c512bc41a3fd4"}
21:40:22 [21:40:22.997]
INFO:
prover-client:proving-broker Proving Broker started
21:40:22 [21:40:22.997]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b4113df4a7943782 err=test error
{"provingJoId":"1:BASE_PARITY:b4113df4a7943782"}
21:40:23 [21:40:23.001]
INFO:
prover-client:proving-broker Proving Broker started
21:40:23 [21:40:23.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e993ffa0ac9a2aca epochNumber=1
{"provingJobId":"1:BASE_PARITY:e993ffa0ac9a2aca"}
21:40:23 [21:40:23.014]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyC1zqo/1 with map size 1073741824
21:40:23 [21:40:23.019]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:23 [21:40:23.043]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e993ffa0ac9a2aca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e993ffa0ac9a2aca"}
21:40:23 [21:40:23.044]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e993ffa0ac9a2aca type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e993ffa0ac9a2aca"}
21:40:23 [21:40:23.063]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:247ce23bdc07b231 epochNumber=2
{"provingJobId":"2:BASE_PARITY:247ce23bdc07b231"}
21:40:23 [21:40:23.065]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyC1zqo/2 with map size 1073741824
21:40:23 [21:40:23.068]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:23 [21:40:23.093]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:247ce23bdc07b231 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:247ce23bdc07b231"}
21:40:23 [21:40:23.094]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:247ce23bdc07b231 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:247ce23bdc07b231"}
21:40:23 [21:40:23.164]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:a03cc80e53443b6a epochNumber=3
{"provingJobId":"3:BASE_PARITY:a03cc80e53443b6a"}
21:40:23 [21:40:23.165]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testyC1zqo/3 with map size 1073741824
21:40:23 [21:40:23.169]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:23 [21:40:23.205]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:23 [21:40:23.206]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:40:23 [21:40:23.251]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:f55bfe2dc92df26d epochNumber=4
{"provingJobId":"4:BASE_PARITY:f55bfe2dc92df26d"}
21:40:23 [21:40:23.253]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testyC1zqo/4 with map size 1073741824
21:40:23 [21:40:23.255]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:23 [21:40:23.286]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:23 [21:40:23.287]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
21:40:23 [21:40:23.335]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:c25ece7e31f57879 epochNumber=5
{"provingJobId":"5:BASE_PARITY:c25ece7e31f57879"}
21:40:23 [21:40:23.337]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testyC1zqo/5 with map size 1073741824
21:40:23 [21:40:23.341]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:40:23 [21:40:23.369]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:40:23 [21:40:23.370]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
21:40:23
PASS src/proving_broker/proving_broker.test.ts (
13.941 s)
21:40:23 ProvingBroker
21:40:23 Producer API
21:40:23
✓ refuses stale jobs (53 ms)
21:40:23
✓ enqueues jobs (4 ms)
21:40:23
✓ ignores duplicate jobs (4 ms)
21:40:23
✓ reports correct status when enqueuing repeat jobs (4 ms)
21:40:23
✓ throws an error in case of duplicate job IDs (36 ms)
21:40:23
✓ returns not-found status for non-existing jobs (3 ms)
21:40:23
✓ cancels jobs in queue (5 ms)
21:40:23
✓ cancels jobs in-progress (6 ms)
21:40:23
✓ returns job result if successful (5 ms)
21:40:23
✓ returns job error if failed (3 ms)
21:40:23
✓ correctly returns job status for concurrent writes (3 ms)
21:40:23
✓ refuses stale jobs (40 ms)
21:40:23
✓ enqueues jobs (49 ms)
21:40:23
✓ ignores duplicate jobs (32 ms)
21:40:23
✓ reports correct status when enqueuing repeat jobs (36 ms)
21:40:23
✓ throws an error in case of duplicate job IDs (43 ms)
21:40:23
✓ returns not-found status for non-existing jobs (3 ms)
21:40:23
✓ cancels jobs in queue (33 ms)
21:40:23
✓ cancels jobs in-progress (39 ms)
21:40:23
✓ returns job result if successful (32 ms)
21:40:23
✓ returns job error if failed (33 ms)
21:40:23
✓ correctly returns job status for concurrent writes (48 ms)
21:40:23 Consumer API
21:40:23
✓ returns undefined if no jobs are available (1 ms)
21:40:23
✓ returns jobs in priority order (2 ms)
21:40:23
✓ returns undefined if no jobs are available for the given allowList (3 ms)
21:40:23
✓ returns a job if it is in the allowList (4 ms)
21:40:23
✓ returns the most important job if it is in the allowList (6 ms)
21:40:23
✓ returns any job if filter is empty (3 ms)
21:40:23
✓ returns a new job when reporting job success (3 ms)
21:40:23
✓ returns a new job when reporting permanent error (3 ms)
21:40:23
✓ returns a new job when reporting retry-able error (2 ms)
21:40:23
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
21:40:23
✓ returns a new job if job is already in progress elsewhere (1061 ms)
21:40:23
✓ avoids sending the same job to a new agent after a restart (1007 ms)
21:40:23
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
21:40:23
✓ tracks job result if in progress (4 ms)
21:40:23
✓ tracks job result even if job is in queue (3 ms)
21:40:23
✓ ignores reported job error if unknown job (2 ms)
21:40:23
✓ ignores job result if unknown job (1 ms)
21:40:23
✓ returns undefined if no jobs are available (1 ms)
21:40:23
✓ returns jobs in priority order (39 ms)
21:40:23
✓ returns undefined if no jobs are available for the given allowList (44 ms)
21:40:23
✓ returns a job if it is in the allowList (62 ms)
21:40:23
✓ returns the most important job if it is in the allowList (68 ms)
21:40:23
✓ returns any job if filter is empty (68 ms)
21:40:23
✓ returns a new job when reporting job success (37 ms)
21:40:23
✓ returns a new job when reporting permanent error (45 ms)
21:40:23
✓ returns a new job when reporting retry-able error (48 ms)
21:40:23
✓ returns a new job when reporting progress if current one is cancelled (40 ms)
21:40:23
✓ returns a new job if job is already in progress elsewhere (1113 ms)
21:40:23
✓ avoids sending the same job to a new agent after a restart (1048 ms)
21:40:23
✓ avoids sending a completed job to a new agent after a restart (1079 ms)
21:40:23
✓ tracks job result if in progress (72 ms)
21:40:23
✓ tracks job result even if job is in queue (63 ms)
21:40:23
✓ ignores reported job error if unknown job (6 ms)
21:40:23
✓ ignores job result if unknown job (4 ms)
21:40:23 Timeouts
21:40:23
✓ tracks in progress jobs (2 ms)
21:40:23
✓ re-enqueues jobs that time out (108 ms)
21:40:23
✓ cancel stale jobs that time out (258 ms)
21:40:23
✓ keeps the jobs in progress while it is alive (205 ms)
21:40:23
✓ tracks in progress jobs (23 ms)
21:40:23
✓ re-enqueues jobs that time out (134 ms)
21:40:23
✓ cancel stale jobs that time out (289 ms)
21:40:23
✓ keeps the jobs in progress while it is alive (228 ms)
21:40:23 Retries
21:40:23
✓ retries jobs (3 ms)
21:40:23
✓ retries up to a maximum number of times (3 ms)
21:40:23
✓ passing retry=false does not retry (1 ms)
21:40:23
✓ does not retry if job is stale (28 ms)
21:40:23
✓ retries jobs (30 ms)
21:40:23
✓ retries up to a maximum number of times (34 ms)
21:40:23
✓ passing retry=false does not retry (34 ms)
21:40:23
✓ does not retry if job is stale (78 ms)
21:40:23 Database management
21:40:23
✓ re-enqueues proof requests on start (3 ms)
21:40:23
✓ restores proof results on start (1 ms)
21:40:23
✓ only re-enqueues unfinished jobs (1 ms)
21:40:23
✓ saves job when enqueued (2 ms)
21:40:23
✓ does not retain job if database fails to save (2 ms)
21:40:23
✓ saves job result (1 ms)
21:40:23
✓ saves result even if database fails to save (2 ms)
21:40:23
✓ saves job error (2 ms)
21:40:23
✓ saves job error even if database fails to save (2 ms)
21:40:23
✓ does not save job result if job is unknown (1 ms)
21:40:23
✓ does not save job error if job is unknown (2 ms)
21:40:23
✓ cleans up old jobs periodically (263 ms)
21:40:23
✓ re-enqueues proof requests on start (50 ms)
21:40:23
✓ restores proof results on start (71 ms)
21:40:23
✓ only re-enqueues unfinished jobs (69 ms)
21:40:23
✓ saves job when enqueued (32 ms)
21:40:23
✓ does not retain job if database fails to save (6 ms)
21:40:23
✓ saves job result (37 ms)
21:40:23
✓ saves result even if database fails to save (29 ms)
21:40:23
✓ saves job error (37 ms)
21:40:23
✓ saves job error even if database fails to save (38 ms)
21:40:23
✓ does not save job result if job is unknown (6 ms)
21:40:23
✓ does not save job error if job is unknown (7 ms)
21:40:23
✓ cleans up old jobs periodically (419 ms)
21:40:23
21:40:23
Test Suites: 1 passed, 1 total
21:40:23
Tests: 96 passed, 96 total
21:40:23
Snapshots: 0 total
21:40:23
Time: 14.052 s
21:40:23
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
21:40:23
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?