Parent Log:
http://ci.aztec-labs.com/103a111326154930
Command: a5511b51e55ea9c0 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/f29a038c58fc1f3ad95ad05d18aea321f0f598b2
Env: REF_NAME=gh-readonly-queue/next/pr-15151-be1fa8aacb33d330cf496352bb1446439e34c51a CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 20 10:14:02 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15151_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
10:14:06 [10:14:06.877]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.884]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87abcce45f8d8bc8 epochNumber=42
{"provingJobId":"1:BASE_PARITY:87abcce45f8d8bc8"}
10:14:06 [10:14:06.887]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:ef0b6da5648c96b5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef0b6da5648c96b5"}
10:14:06 [10:14:06.930]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.930]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71652a8ae3932d12 epochNumber=1
{"provingJobId":"1:BASE_PARITY:71652a8ae3932d12"}
10:14:06 [10:14:06.932]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:781434664bf45008 epochNumber=1
{"provingJobId":"1:BASE_PARITY:781434664bf45008"}
10:14:06 [10:14:06.934]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.934]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b5801fedc89ca3a epochNumber=1
{"provingJobId":"1:BASE_PARITY:5b5801fedc89ca3a"}
10:14:06 [10:14:06.936]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5b5801fedc89ca3a epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5b5801fedc89ca3a"}
10:14:06 [10:14:06.938]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5418b4a7ac5ef822 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5418b4a7ac5ef822"}
10:14:06 [10:14:06.940]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5418b4a7ac5ef822 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5418b4a7ac5ef822"}
10:14:06 [10:14:06.941]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5418b4a7ac5ef822 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5418b4a7ac5ef822"}
10:14:06 [10:14:06.942]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5418b4a7ac5ef822 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5418b4a7ac5ef822"}
10:14:06 [10:14:06.943]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.943]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:63ffe7d9c8b9ce69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:63ffe7d9c8b9ce69"}
10:14:06 [10:14:06.975]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.977]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.978]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f940e09068becf0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f940e09068becf0"}
10:14:06 [10:14:06.978]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3f940e09068becf0
{"provingJobId":"1:BASE_PARITY:3f940e09068becf0"}
10:14:06 [10:14:06.979]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3f940e09068becf0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3f940e09068becf0"}
10:14:06 [10:14:06.979]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3f940e09068becf0 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3f940e09068becf0"}
10:14:06 [10:14:06.981]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.981]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09783febf336485b epochNumber=1
{"provingJobId":"1:BASE_PARITY:09783febf336485b"}
10:14:06 [10:14:06.982]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:09783febf336485b
{"provingJobId":"1:BASE_PARITY:09783febf336485b"}
10:14:06 [10:14:06.982]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:09783febf336485b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:09783febf336485b"}
10:14:06 [10:14:06.993]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.994]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:498d314537ecec57 epochNumber=1
{"provingJobId":"1:BASE_PARITY:498d314537ecec57"}
10:14:06 [10:14:06.995]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:498d314537ecec57 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:498d314537ecec57"}
10:14:06 [10:14:06.995]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:498d314537ecec57 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:498d314537ecec57"}
10:14:06 [10:14:06.997]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:06.998]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2450c1e77df055d epochNumber=1
{"provingJobId":"1:BASE_PARITY:c2450c1e77df055d"}
10:14:06 [10:14:06.998]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c2450c1e77df055d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c2450c1e77df055d"}
10:14:06 [10:14:06.998]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c2450c1e77df055d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c2450c1e77df055d"}
10:14:06 [10:14:06.999]
INFO:
prover-client:proving-broker Proving Broker started
10:14:06 [10:14:07.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b2a4f089552d378 epochNumber=0
{"provingJobId":"1:BASE_PARITY:2b2a4f089552d378"}
10:14:07 [10:14:07.000]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2b2a4f089552d378 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2b2a4f089552d378"}
10:14:07 [10:14:07.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05d9641859fb1c5c epochNumber=0
{"provingJobId":"1:BASE_PARITY:05d9641859fb1c5c"}
10:14:07 [10:14:07.001]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2b2a4f089552d378 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2b2a4f089552d378"}
10:14:07 [10:14:07.001]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9185d102b7e4d23f epochNumber=0
{"provingJobId":"1:BASE_PARITY:9185d102b7e4d23f"}
10:14:07 [10:14:07.002]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.004]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.005]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c7d4afe25fa020c epochNumber=2
{"provingJobId":"1:BASE_PARITY:6c7d4afe25fa020c"}
10:14:07 [10:14:07.006]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae8263294308b3b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae8263294308b3b4"}
10:14:07 [10:14:07.007]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.007]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:feb75d9b38943d03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:feb75d9b38943d03"}
10:14:07 [10:14:07.008]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.009]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e53090e928c9b369 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e53090e928c9b369"}
10:14:07 [10:14:07.009]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e79c95f3fdd27c32 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e79c95f3fdd27c32"}
10:14:07 [10:14:07.009]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c821a13a0b57a551 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c821a13a0b57a551"}
10:14:07 [10:14:07.010]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:63abe2b5b1b70889 epochNumber=1
{"provingJobId":"1:BASE_PARITY:63abe2b5b1b70889"}
10:14:07 [10:14:07.011]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.011]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:64dcf54ee72ed03b epochNumber=1
{"provingJobId":"1:BASE_PARITY:64dcf54ee72ed03b"}
10:14:07 [10:14:07.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00d2a827f43a16c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:00d2a827f43a16c9"}
10:14:07 [10:14:07.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f499c4cd9541bc59 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f499c4cd9541bc59"}
10:14:07 [10:14:07.012]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a07cb53b61874bec epochNumber=1
{"provingJobId":"1:BASE_PARITY:a07cb53b61874bec"}
10:14:07 [10:14:07.013]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.014]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94e4685e81a89674 epochNumber=1
{"provingJobId":"1:BASE_PARITY:94e4685e81a89674"}
10:14:07 [10:14:07.014]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52b2059ec46f08dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:52b2059ec46f08dd"}
10:14:07 [10:14:07.015]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89bac15a9bbfb739 epochNumber=2
{"provingJobId":"1:BASE_PARITY:89bac15a9bbfb739"}
10:14:07 [10:14:07.015]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f53a084fca4787c epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f53a084fca4787c"}
10:14:07 [10:14:07.016]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.017]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b82373f7253d50f epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b82373f7253d50f"}
10:14:07 [10:14:07.017]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25bfda1c675c979a epochNumber=1
{"provingJobId":"1:BASE_PARITY:25bfda1c675c979a"}
10:14:07 [10:14:07.018]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6b82373f7253d50f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6b82373f7253d50f"}
10:14:07 [10:14:07.019]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c65f23f81f32257 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c65f23f81f32257"}
10:14:07 [10:14:07.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47016504a59d26ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:47016504a59d26ed"}
10:14:07 [10:14:07.020]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3c65f23f81f32257 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:3c65f23f81f32257"}
10:14:07 [10:14:07.021]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.022]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0336a13327a3d8fc epochNumber=1
{"provingJobId":"1:BASE_PARITY:0336a13327a3d8fc"}
10:14:07 [10:14:07.022]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:056bfea5a58f76e9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:056bfea5a58f76e9"}
10:14:07 [10:14:07.023]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0336a13327a3d8fc type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:0336a13327a3d8fc"}
10:14:07 [10:14:07.024]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.024]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:37c5ba0097a87846 epochNumber=1
{"provingJobId":"1:BASE_PARITY:37c5ba0097a87846"}
10:14:07 [10:14:07.025]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:37c5ba0097a87846
{"provingJobId":"1:BASE_PARITY:37c5ba0097a87846"}
10:14:07 [10:14:07.025]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:37c5ba0097a87846 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:37c5ba0097a87846"}
10:14:07 [10:14:07.026]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b29943428cabfbf epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b29943428cabfbf"}
10:14:07 [10:14:07.026]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:37c5ba0097a87846 has already been completed
{"provingJobId":"1:BASE_PARITY:37c5ba0097a87846"}
10:14:07 [10:14:07.027]
INFO:
prover-client:proving-broker Proving Broker started
10:14:07 [10:14:07.027]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1590c86a189c6fb8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1590c86a189c6fb8"}
10:14:07 [10:14:07.028]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b3b77f2ecd713ad epochNumber=2
{"provingJobId":"1:BASE_PARITY:2b3b77f2ecd713ad"}
10:14:08 [10:14:08.080]
INFO:
prover-client:proving-broker Proving Broker started
10:14:08 [10:14:08.081]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1590c86a189c6fb8 settled=false
{"provingJobId":"1:BASE_PARITY:1590c86a189c6fb8","status":"pending"}
10:14:08 [10:14:08.081]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2b3b77f2ecd713ad settled=false
{"provingJobId":"1:BASE_PARITY:2b3b77f2ecd713ad","status":"pending"}
10:14:08 [10:14:08.082]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1590c86a189c6fb8 type=BASE_PARITY startedAt=1750414447028 older agent has taken job
{"provingJobId":"1:BASE_PARITY:1590c86a189c6fb8"}
10:14:08 [10:14:08.084]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1590c86a189c6fb8 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:1590c86a189c6fb8"}
10:14:08 [10:14:08.087]
INFO:
prover-client:proving-broker Proving Broker started
10:14:08 [10:14:08.088]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d21dcf56b5fca45e epochNumber=1
{"provingJobId":"1:BASE_PARITY:d21dcf56b5fca45e"}
10:14:08 [10:14:08.089]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f32154339a3290fc epochNumber=2
{"provingJobId":"1:BASE_PARITY:f32154339a3290fc"}
10:14:09 [10:14:09.090]
INFO:
prover-client:proving-broker Proving Broker started
10:14:09 [10:14:09.090]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d21dcf56b5fca45e settled=false
{"provingJobId":"1:BASE_PARITY:d21dcf56b5fca45e","status":"pending"}
10:14:09 [10:14:09.091]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f32154339a3290fc settled=false
{"provingJobId":"1:BASE_PARITY:f32154339a3290fc","status":"pending"}
10:14:09 [10:14:09.092]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d21dcf56b5fca45e type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:d21dcf56b5fca45e"}
10:14:09 [10:14:09.093]
INFO:
prover-client:proving-broker Proving Broker started
10:14:09 [10:14:09.094]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b8f7378e16e9b74f epochNumber=1
{"provingJobId":"1:BASE_PARITY:b8f7378e16e9b74f"}
10:14:09 [10:14:09.095]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ccdfaf493b7eda5e epochNumber=2
{"provingJobId":"1:BASE_PARITY:ccdfaf493b7eda5e"}
10:14:10 [10:14:10.095]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.095]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b8f7378e16e9b74f settled=false
{"provingJobId":"1:BASE_PARITY:b8f7378e16e9b74f","status":"pending"}
10:14:10 [10:14:10.096]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ccdfaf493b7eda5e settled=false
{"provingJobId":"1:BASE_PARITY:ccdfaf493b7eda5e","status":"pending"}
10:14:10 [10:14:10.097]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b8f7378e16e9b74f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b8f7378e16e9b74f"}
10:14:10 [10:14:10.097]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b8f7378e16e9b74f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b8f7378e16e9b74f"}
10:14:10 [10:14:10.098]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.099]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc2acb4a1df74cbc epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc2acb4a1df74cbc"}
10:14:10 [10:14:10.099]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:921d770b52d1257d epochNumber=2
{"provingJobId":"1:BASE_PARITY:921d770b52d1257d"}
10:14:10 [10:14:10.099]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cc2acb4a1df74cbc type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cc2acb4a1df74cbc"}
10:14:10 [10:14:10.100]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:921d770b52d1257d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:921d770b52d1257d"}
10:14:10 [10:14:10.101]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.102]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:013e5db6083d6184 epochNumber=1
{"provingJobId":"1:BASE_PARITY:013e5db6083d6184"}
10:14:10 [10:14:10.102]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:73d93db80817c3e3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:73d93db80817c3e3"}
10:14:10 [10:14:10.102]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:013e5db6083d6184 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:013e5db6083d6184"}
10:14:10 [10:14:10.103]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:013e5db6083d6184 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:013e5db6083d6184"}
10:14:10 [10:14:10.103]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:73d93db80817c3e3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:73d93db80817c3e3"}
10:14:10 [10:14:10.104]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.105]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:61d2e6aa38da5de6 err=test error
{"provingJoId":"1:BASE_PARITY:61d2e6aa38da5de6"}
10:14:10 [10:14:10.105]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.106]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f9cc94eb09b7d868 not found
{"provingJobId":"1:BASE_PARITY:f9cc94eb09b7d868"}
10:14:10 [10:14:10.108]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.108]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ae319d4faa1075f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ae319d4faa1075f"}
10:14:10 [10:14:10.110]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.111]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3626562323206d76 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3626562323206d76"}
10:14:10 [10:14:10.213]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3626562323206d76 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:3626562323206d76"}
10:14:10 [10:14:10.221]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34316db273cbd6a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:34316db273cbd6a0"}
10:14:10 [10:14:10.325]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:34316db273cbd6a0 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:34316db273cbd6a0"}
10:14:10 [10:14:10.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed0edc902cca6434 epochNumber=10
{"provingJobId":"1:BASE_PARITY:ed0edc902cca6434"}
10:14:10 [10:14:10.351]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:10 [10:14:10.457]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.458]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cafef54581abab90 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cafef54581abab90"}
10:14:10 [10:14:10.612]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cafef54581abab90 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:cafef54581abab90"}
10:14:10 [10:14:10.662]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.663]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba542cae84ee3ca8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba542cae84ee3ca8"}
10:14:10 [10:14:10.664]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ba542cae84ee3ca8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:ba542cae84ee3ca8"}
10:14:10 [10:14:10.666]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.666]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2666fea5da5bcf22 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2666fea5da5bcf22"}
10:14:10 [10:14:10.667]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2666fea5da5bcf22 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:2666fea5da5bcf22"}
10:14:10 [10:14:10.667]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2666fea5da5bcf22 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:2666fea5da5bcf22"}
10:14:10 [10:14:10.669]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.669]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef3b01e8523890f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef3b01e8523890f8"}
10:14:10 [10:14:10.669]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ef3b01e8523890f8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ef3b01e8523890f8"}
10:14:10 [10:14:10.673]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.674]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0df1054696894d06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0df1054696894d06"}
10:14:10 [10:14:10.675]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0df1054696894d06 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0df1054696894d06"}
10:14:10 [10:14:10.676]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5483ec2a09668024 epochNumber=3
{"provingJobId":"1:BASE_PARITY:5483ec2a09668024"}
10:14:10 [10:14:10.699]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:10 [10:14:10.702]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0df1054696894d06 err=test error
{"provingJoId":"1:BASE_PARITY:0df1054696894d06"}
10:14:10 [10:14:10.704]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.705]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:553cd09546c0f1e1 settled=false
{"provingJobId":"1:BASE_PARITY:553cd09546c0f1e1","status":"pending"}
10:14:10 [10:14:10.705]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:edb419929cd15c40 settled=false
{"provingJobId":"1:BASE_PARITY:edb419929cd15c40","status":"pending"}
10:14:10 [10:14:10.707]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.707]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:00285d559e5fcd29 settled=true
{"provingJobId":"1:BASE_PARITY:00285d559e5fcd29","status":"fulfilled"}
10:14:10 [10:14:10.708]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:c93c19b9ecbf289d settled=true
{"provingJobId":"2:BASE_PARITY:c93c19b9ecbf289d","status":"fulfilled"}
10:14:10 [10:14:10.709]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.709]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c8e261b506e89461 settled=true
{"provingJobId":"1:BASE_PARITY:c8e261b506e89461","status":"fulfilled"}
10:14:10 [10:14:10.709]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cf4583eaab736eef settled=false
{"provingJobId":"1:BASE_PARITY:cf4583eaab736eef","status":"pending"}
10:14:10 [10:14:10.710]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.712]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c7439db86f4b4042 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c7439db86f4b4042"}
10:14:10 [10:14:10.714]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.714]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:39dc887572e5241a epochNumber=1
{"provingJobId":"1:BASE_PARITY:39dc887572e5241a"}
10:14:10 [10:14:10.715]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:39dc887572e5241a: Error: db error: Error: db error
10:14:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:39dc887572e5241a"}
10:14:10 [10:14:10.716]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c179e76bc19555a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c179e76bc19555a1"}
10:14:10 [10:14:10.717]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c179e76bc19555a1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c179e76bc19555a1"}
10:14:10 [10:14:10.717]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c179e76bc19555a1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c179e76bc19555a1"}
10:14:10 [10:14:10.718]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.719]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9943fa8c45999aba epochNumber=1
{"provingJobId":"1:BASE_PARITY:9943fa8c45999aba"}
10:14:10 [10:14:10.719]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9943fa8c45999aba type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9943fa8c45999aba"}
10:14:10 [10:14:10.719]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9943fa8c45999aba type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9943fa8c45999aba"}
10:14:10 [10:14:10.720]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:9943fa8c45999aba: Error: db error
10:14:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:9943fa8c45999aba"}
10:14:10 [10:14:10.721]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.721]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6ac4b07ad55a985 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d6ac4b07ad55a985"}
10:14:10 [10:14:10.721]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d6ac4b07ad55a985 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d6ac4b07ad55a985"}
10:14:10 [10:14:10.722]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d6ac4b07ad55a985 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d6ac4b07ad55a985"}
10:14:10 [10:14:10.723]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.723]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e05abc6b3aa9ae4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e05abc6b3aa9ae4"}
10:14:10 [10:14:10.723]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5e05abc6b3aa9ae4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5e05abc6b3aa9ae4"}
10:14:10 [10:14:10.723]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5e05abc6b3aa9ae4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5e05abc6b3aa9ae4"}
10:14:10 [10:14:10.724]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:5e05abc6b3aa9ae4 jobErr=test error: Error: db error
10:14:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:5e05abc6b3aa9ae4"}
10:14:10 [10:14:10.725]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.725]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dcc9b2cb0f22333c not found
{"provingJobId":"1:BASE_PARITY:dcc9b2cb0f22333c"}
10:14:10 [10:14:10.726]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.727]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ea652dc52ef85342 err=test error
{"provingJoId":"1:BASE_PARITY:ea652dc52ef85342"}
10:14:10 [10:14:10.728]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.738]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:936a2223f86f789b epochNumber=1
{"provingJobId":"1:BASE_PARITY:936a2223f86f789b"}
10:14:10 [10:14:10.738]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:936a2223f86f789b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:936a2223f86f789b"}
10:14:10 [10:14:10.739]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:936a2223f86f789b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:936a2223f86f789b"}
10:14:10 [10:14:10.749]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:2abcf05b37515539 epochNumber=2
{"provingJobId":"2:BASE_PARITY:2abcf05b37515539"}
10:14:10 [10:14:10.749]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:2abcf05b37515539 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:2abcf05b37515539"}
10:14:10 [10:14:10.749]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2abcf05b37515539 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:2abcf05b37515539"}
10:14:10 [10:14:10.811]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:887649e28650a330 epochNumber=3
{"provingJobId":"3:BASE_PARITY:887649e28650a330"}
10:14:10 [10:14:10.829]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:10 [10:14:10.873]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:02e907df4feca3c6 epochNumber=4
{"provingJobId":"4:BASE_PARITY:02e907df4feca3c6"}
10:14:10 [10:14:10.884]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:10 [10:14:10.935]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:b5c74c4239038733 epochNumber=5
{"provingJobId":"5:BASE_PARITY:b5c74c4239038733"}
10:14:10 [10:14:10.961]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:10 [10:14:10.990]
INFO:
prover-client:proving-broker Proving Broker started
10:14:10 [10:14:10.990]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb003d6a76b92f41 epochNumber=42
{"provingJobId":"1:BASE_PARITY:bb003d6a76b92f41"}
10:14:10 [10:14:10.991]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testXewRla/42 with map size 1073741824
10:14:10 [10:14:10.995]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.024]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:c17f241f909c7dab epochNumber=1
{"provingJobId":"1:BASE_PARITY:c17f241f909c7dab"}
10:14:11 [10:14:11.027]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.028]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7045234ee1d32b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7045234ee1d32b3"}
10:14:11 [10:14:11.029]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJ0JuSt/1 with map size 1073741824
10:14:11 [10:14:11.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.055]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f411f904fdc4922 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f411f904fdc4922"}
10:14:11 [10:14:11.069]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.070]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87d40262cf8c0f63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87d40262cf8c0f63"}
10:14:11 [10:14:11.071]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUrjULQ/1 with map size 1073741824
10:14:11 [10:14:11.077]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.098]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:87d40262cf8c0f63 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:87d40262cf8c0f63"}
10:14:11 [10:14:11.102]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.102]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d21ba2f9127e2230 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d21ba2f9127e2230"}
10:14:11 [10:14:11.103]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfcCD8l/1 with map size 1073741824
10:14:11 [10:14:11.105]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.124]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d21ba2f9127e2230 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d21ba2f9127e2230"}
10:14:11 [10:14:11.125]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d21ba2f9127e2230 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d21ba2f9127e2230"}
10:14:11 [10:14:11.131]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d21ba2f9127e2230 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d21ba2f9127e2230"}
10:14:11 [10:14:11.137]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.137]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2bdf2b6409ca8b9d epochNumber=1
{"provingJobId":"1:BASE_PARITY:2bdf2b6409ca8b9d"}
10:14:11 [10:14:11.138]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6Fb7Kx/1 with map size 1073741824
10:14:11 [10:14:11.140]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.160]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.163]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.163]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f66d9639a7767740 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f66d9639a7767740"}
10:14:11 [10:14:11.164]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvhC03D/1 with map size 1073741824
10:14:11 [10:14:11.172]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.192]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f66d9639a7767740
{"provingJobId":"1:BASE_PARITY:f66d9639a7767740"}
10:14:11 [10:14:11.192]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f66d9639a7767740 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f66d9639a7767740"}
10:14:11 [10:14:11.193]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f66d9639a7767740 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f66d9639a7767740"}
10:14:11 [10:14:11.202]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.203]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7cf06fe25e6b296c epochNumber=1
{"provingJobId":"1:BASE_PARITY:7cf06fe25e6b296c"}
10:14:11 [10:14:11.204]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrgH1nN/1 with map size 1073741824
10:14:11 [10:14:11.206]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.224]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7cf06fe25e6b296c
{"provingJobId":"1:BASE_PARITY:7cf06fe25e6b296c"}
10:14:11 [10:14:11.224]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7cf06fe25e6b296c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7cf06fe25e6b296c"}
10:14:11 [10:14:11.234]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc1e61a88ffc95b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc1e61a88ffc95b8"}
10:14:11 [10:14:11.236]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTTWHex/1 with map size 1073741824
10:14:11 [10:14:11.239]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.256]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fc1e61a88ffc95b8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fc1e61a88ffc95b8"}
10:14:11 [10:14:11.256]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fc1e61a88ffc95b8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fc1e61a88ffc95b8"}
10:14:11 [10:14:11.269]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.270]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12e56ce4c2f779c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:12e56ce4c2f779c8"}
10:14:11 [10:14:11.270]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1MxBi4/1 with map size 1073741824
10:14:11 [10:14:11.272]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.291]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:12e56ce4c2f779c8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:12e56ce4c2f779c8"}
10:14:11 [10:14:11.292]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:12e56ce4c2f779c8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:12e56ce4c2f779c8"}
10:14:11 [10:14:11.300]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.302]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3d51fe670cf8060 epochNumber=0
{"provingJobId":"1:BASE_PARITY:f3d51fe670cf8060"}
10:14:11 [10:14:11.303]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testjS3hTK/0 with map size 1073741824
10:14:11 [10:14:11.305]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.323]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f3d51fe670cf8060 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f3d51fe670cf8060"}
10:14:11 [10:14:11.324]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3cca244dab9748f3 epochNumber=0
{"provingJobId":"1:BASE_PARITY:3cca244dab9748f3"}
10:14:11 [10:14:11.325]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f3d51fe670cf8060 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:f3d51fe670cf8060"}
10:14:11 [10:14:11.325]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:912861e6d279cfd7 epochNumber=0
{"provingJobId":"1:BASE_PARITY:912861e6d279cfd7"}
10:14:11 [10:14:11.341]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.343]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.344]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84d6b17d9ec54ae0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:84d6b17d9ec54ae0"}
10:14:11 [10:14:11.345]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test9bPsTA/2 with map size 1073741824
10:14:11 [10:14:11.347]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7dbd0166855da129 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7dbd0166855da129"}
10:14:11 [10:14:11.366]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9bPsTA/1 with map size 1073741824
10:14:11 [10:14:11.370]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.391]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.392]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:732e54bbfe49b5a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:732e54bbfe49b5a7"}
10:14:11 [10:14:11.392]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMsop2e/1 with map size 1073741824
10:14:11 [10:14:11.394]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.412]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.412]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c95727440cdedc03 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c95727440cdedc03"}
10:14:11 [10:14:11.413]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKekmoj/1 with map size 1073741824
10:14:11 [10:14:11.416]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.433]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:adc42b30464bf84d epochNumber=1
{"provingJobId":"1:BASE_PARITY:adc42b30464bf84d"}
10:14:11 [10:14:11.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7603f025a5ba8829 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7603f025a5ba8829"}
10:14:11 [10:14:11.440]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKekmoj/2 with map size 1073741824
10:14:11 [10:14:11.442]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c2beef92344477d epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c2beef92344477d"}
10:14:11 [10:14:11.473]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.474]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3519ba33151ec78c epochNumber=1
{"provingJobId":"1:BASE_PARITY:3519ba33151ec78c"}
10:14:11 [10:14:11.478]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testejS779/1 with map size 1073741824
10:14:11 [10:14:11.481]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0df2ba7ba4a47b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0df2ba7ba4a47b9"}
10:14:11 [10:14:11.507]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4d78e93a29372f7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d4d78e93a29372f7"}
10:14:11 [10:14:11.508]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testejS779/2 with map size 1073741824
10:14:11 [10:14:11.510]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.526]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55f9bb28217d4ac5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:55f9bb28217d4ac5"}
10:14:11 [10:14:11.536]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.536]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cfd132d49d88b802 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cfd132d49d88b802"}
10:14:11 [10:14:11.537]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testr8oZea/1 with map size 1073741824
10:14:11 [10:14:11.539]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.555]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46b53d8743410743 epochNumber=1
{"provingJobId":"1:BASE_PARITY:46b53d8743410743"}
10:14:11 [10:14:11.561]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e67206e963dcc7da epochNumber=2
{"provingJobId":"1:BASE_PARITY:e67206e963dcc7da"}
10:14:11 [10:14:11.562]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testr8oZea/2 with map size 1073741824
10:14:11 [10:14:11.568]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.586]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48a988eef46a04cc epochNumber=1
{"provingJobId":"1:BASE_PARITY:48a988eef46a04cc"}
10:14:11 [10:14:11.595]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.596]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f55e1fe8267abb5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f55e1fe8267abb5"}
10:14:11 [10:14:11.597]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8zqeyk/1 with map size 1073741824
10:14:11 [10:14:11.598]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.615]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d25ef816fb52bcae epochNumber=1
{"provingJobId":"1:BASE_PARITY:d25ef816fb52bcae"}
10:14:11 [10:14:11.621]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4f55e1fe8267abb5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4f55e1fe8267abb5"}
10:14:11 [10:14:11.630]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.630]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d795744cf14e07b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d795744cf14e07b8"}
10:14:11 [10:14:11.631]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test49fTbb/1 with map size 1073741824
10:14:11 [10:14:11.632]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.650]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50654373b48afb0a epochNumber=1
{"provingJobId":"1:BASE_PARITY:50654373b48afb0a"}
10:14:11 [10:14:11.656]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d795744cf14e07b8 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:d795744cf14e07b8"}
10:14:11 [10:14:11.666]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.667]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aca5847a28d6f80d epochNumber=1
{"provingJobId":"1:BASE_PARITY:aca5847a28d6f80d"}
10:14:11 [10:14:11.667]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSyrmai/1 with map size 1073741824
10:14:11 [10:14:11.671]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82f1b65f916d38fc epochNumber=1
{"provingJobId":"1:BASE_PARITY:82f1b65f916d38fc"}
10:14:11 [10:14:11.701]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:aca5847a28d6f80d type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:aca5847a28d6f80d"}
10:14:11 [10:14:11.705]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.706]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c964cb75bfeff12e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c964cb75bfeff12e"}
10:14:11 [10:14:11.706]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testR6DSyg/1 with map size 1073741824
10:14:11 [10:14:11.708]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.724]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c964cb75bfeff12e
{"provingJobId":"1:BASE_PARITY:c964cb75bfeff12e"}
10:14:11 [10:14:11.724]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c964cb75bfeff12e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c964cb75bfeff12e"}
10:14:11 [10:14:11.730]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ea0fbfc99d54b80 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ea0fbfc99d54b80"}
10:14:11 [10:14:11.737]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c964cb75bfeff12e has already been completed
{"provingJobId":"1:BASE_PARITY:c964cb75bfeff12e"}
10:14:11 [10:14:11.740]
INFO:
prover-client:proving-broker Proving Broker started
10:14:11 [10:14:11.741]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f02d59e8db538dc8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f02d59e8db538dc8"}
10:14:11 [10:14:11.744]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXvvhjS/1 with map size 1073741824
10:14:11 [10:14:11.747]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:11 [10:14:11.768]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:956b6f440a6fbbd3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:956b6f440a6fbbd3"}
10:14:11 [10:14:11.770]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testXvvhjS/2 with map size 1073741824
10:14:11 [10:14:11.775]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:12 [10:14:12.847]
INFO:
prover-client:proving-broker Proving Broker started
10:14:12 [10:14:12.855]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f02d59e8db538dc8 settled=false
{"provingJobId":"1:BASE_PARITY:f02d59e8db538dc8","status":"pending"}
10:14:12 [10:14:12.856]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:956b6f440a6fbbd3 settled=false
{"provingJobId":"1:BASE_PARITY:956b6f440a6fbbd3","status":"pending"}
10:14:12 [10:14:12.857]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f02d59e8db538dc8 type=BASE_PARITY startedAt=1750414451794 older agent has taken job
{"provingJobId":"1:BASE_PARITY:f02d59e8db538dc8"}
10:14:12 [10:14:12.858]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f02d59e8db538dc8 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:f02d59e8db538dc8"}
10:14:12 [10:14:12.862]
INFO:
prover-client:proving-broker Proving Broker started
10:14:12 [10:14:12.864]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b01bd32d6ca5c66b epochNumber=1
{"provingJobId":"1:BASE_PARITY:b01bd32d6ca5c66b"}
10:14:12 [10:14:12.867]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJ6hEe8/1 with map size 1073741824
10:14:12 [10:14:12.874]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:12 [10:14:12.895]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd81e1b8f43abb8d epochNumber=2
{"provingJobId":"1:BASE_PARITY:dd81e1b8f43abb8d"}
10:14:12 [10:14:12.896]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJ6hEe8/2 with map size 1073741824
10:14:12 [10:14:12.899]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:13 [10:14:13.923]
INFO:
prover-client:proving-broker Proving Broker started
10:14:13 [10:14:13.925]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b01bd32d6ca5c66b settled=false
{"provingJobId":"1:BASE_PARITY:b01bd32d6ca5c66b","status":"pending"}
10:14:13 [10:14:13.926]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dd81e1b8f43abb8d settled=false
{"provingJobId":"1:BASE_PARITY:dd81e1b8f43abb8d","status":"pending"}
10:14:13 [10:14:13.927]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b01bd32d6ca5c66b type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:b01bd32d6ca5c66b"}
10:14:13 [10:14:13.934]
INFO:
prover-client:proving-broker Proving Broker started
10:14:13 [10:14:13.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:795956ae0ce44b65 epochNumber=1
{"provingJobId":"1:BASE_PARITY:795956ae0ce44b65"}
10:14:13 [10:14:13.935]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMITrq7/1 with map size 1073741824
10:14:13 [10:14:13.938]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:13 [10:14:13.960]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d85fe14bb2a6bdfa epochNumber=2
{"provingJobId":"1:BASE_PARITY:d85fe14bb2a6bdfa"}
10:14:13 [10:14:13.961]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMITrq7/2 with map size 1073741824
10:14:13 [10:14:13.963]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:14 [10:14:14.985]
INFO:
prover-client:proving-broker Proving Broker started
10:14:14 [10:14:14.987]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:795956ae0ce44b65 settled=false
{"provingJobId":"1:BASE_PARITY:795956ae0ce44b65","status":"pending"}
10:14:14 [10:14:14.989]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d85fe14bb2a6bdfa settled=false
{"provingJobId":"1:BASE_PARITY:d85fe14bb2a6bdfa","status":"pending"}
10:14:14 [10:14:14.990]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:795956ae0ce44b65 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:795956ae0ce44b65"}
10:14:14 [10:14:14.990]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:795956ae0ce44b65 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:795956ae0ce44b65"}
10:14:15 [10:14:15.003]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.004]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8fa6148c23393302 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8fa6148c23393302"}
10:14:15 [10:14:15.005]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6ZoGFs/1 with map size 1073741824
10:14:15 [10:14:15.006]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e286c34478de5523 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e286c34478de5523"}
10:14:15 [10:14:15.025]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6ZoGFs/2 with map size 1073741824
10:14:15 [10:14:15.027]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.045]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8fa6148c23393302 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8fa6148c23393302"}
10:14:15 [10:14:15.052]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e286c34478de5523 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e286c34478de5523"}
10:14:15 [10:14:15.062]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f733b5a19bfc4f6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f733b5a19bfc4f6"}
10:14:15 [10:14:15.063]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCfUmGa/1 with map size 1073741824
10:14:15 [10:14:15.065]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c8ccc23dc64e905f epochNumber=2
{"provingJobId":"1:BASE_PARITY:c8ccc23dc64e905f"}
10:14:15 [10:14:15.083]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCfUmGa/2 with map size 1073741824
10:14:15 [10:14:15.085]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.103]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6f733b5a19bfc4f6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6f733b5a19bfc4f6"}
10:14:15 [10:14:15.104]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6f733b5a19bfc4f6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6f733b5a19bfc4f6"}
10:14:15 [10:14:15.110]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c8ccc23dc64e905f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c8ccc23dc64e905f"}
10:14:15 [10:14:15.120]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.122]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f9f3d0c764b24a16 err=test error
{"provingJoId":"1:BASE_PARITY:f9f3d0c764b24a16"}
10:14:15 [10:14:15.124]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.125]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b4eb228886e88fb7 not found
{"provingJobId":"1:BASE_PARITY:b4eb228886e88fb7"}
10:14:15 [10:14:15.128]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.129]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:450240f18e2c19e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:450240f18e2c19e7"}
10:14:15 [10:14:15.130]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testc6gaZC/1 with map size 1073741824
10:14:15 [10:14:15.131]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.151]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.151]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:514522679c276c73 epochNumber=1
{"provingJobId":"1:BASE_PARITY:514522679c276c73"}
10:14:15 [10:14:15.152]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testk8tiZc/1 with map size 1073741824
10:14:15 [10:14:15.153]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.277]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:514522679c276c73 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:514522679c276c73"}
10:14:15 [10:14:15.287]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.287]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a2eaee200d0ebb6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a2eaee200d0ebb6"}
10:14:15 [10:14:15.288]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2X3Egt/1 with map size 1073741824
10:14:15 [10:14:15.290]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.420]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5a2eaee200d0ebb6 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:5a2eaee200d0ebb6"}
10:14:15 [10:14:15.423]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:892244f31443f432 epochNumber=10
{"provingJobId":"1:BASE_PARITY:892244f31443f432"}
10:14:15 [10:14:15.424]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test2X3Egt/10 with map size 1073741824
10:14:15 [10:14:15.426]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.446]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:15 [10:14:15.447]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:14:15 [10:14:15.586]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.587]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bba3087ab88792ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:bba3087ab88792ed"}
10:14:15 [10:14:15.588]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXUhMf0/1 with map size 1073741824
10:14:15 [10:14:15.591]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.766]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bba3087ab88792ed timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:bba3087ab88792ed"}
10:14:15 [10:14:15.819]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.820]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0080f5c09d2d7860 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0080f5c09d2d7860"}
10:14:15 [10:14:15.824]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuY1xug/1 with map size 1073741824
10:14:15 [10:14:15.826]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.849]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0080f5c09d2d7860 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0080f5c09d2d7860"}
10:14:15 [10:14:15.855]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.856]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5c995fa01f80b82 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5c995fa01f80b82"}
10:14:15 [10:14:15.857]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVR9DoX/1 with map size 1073741824
10:14:15 [10:14:15.860]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.889]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b5c995fa01f80b82 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b5c995fa01f80b82"}
10:14:15 [10:14:15.890]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b5c995fa01f80b82 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:b5c995fa01f80b82"}
10:14:15 [10:14:15.906]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.907]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:229388989b961047 epochNumber=1
{"provingJobId":"1:BASE_PARITY:229388989b961047"}
10:14:15 [10:14:15.909]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRBI5VS/1 with map size 1073741824
10:14:15 [10:14:15.914]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.935]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:229388989b961047 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:229388989b961047"}
10:14:15 [10:14:15.947]
INFO:
prover-client:proving-broker Proving Broker started
10:14:15 [10:14:15.948]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:302c8705c53a930c epochNumber=1
{"provingJobId":"1:BASE_PARITY:302c8705c53a930c"}
10:14:15 [10:14:15.949]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtCeVcu/1 with map size 1073741824
10:14:15 [10:14:15.951]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:15 [10:14:15.983]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:302c8705c53a930c type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:302c8705c53a930c"}
10:14:15 [10:14:15.985]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d1b01e079e5ed37 epochNumber=3
{"provingJobId":"1:BASE_PARITY:6d1b01e079e5ed37"}
10:14:15 [10:14:15.986]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testtCeVcu/3 with map size 1073741824
10:14:15 [10:14:15.990]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.023]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:16 [10:14:16.024]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:14:16 [10:14:16.035]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:302c8705c53a930c err=test error
{"provingJoId":"1:BASE_PARITY:302c8705c53a930c"}
10:14:16 [10:14:16.042]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOMReYI/1 with map size 1073741824
10:14:16 [10:14:16.045]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.066]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOMReYI/2 with map size 1073741824
10:14:16 [10:14:16.081]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.099]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.101]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:48085c5fca232d21 settled=false
{"provingJobId":"1:BASE_PARITY:48085c5fca232d21","status":"pending"}
10:14:16 [10:14:16.103]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:47a4706d3b4c0b02 settled=false
{"provingJobId":"1:BASE_PARITY:47a4706d3b4c0b02","status":"pending"}
10:14:16 [10:14:16.114]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testie2q64/1 with map size 1073741824
10:14:16 [10:14:16.117]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.136]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testie2q64/2 with map size 1073741824
10:14:16 [10:14:16.138]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.170]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.174]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:993666f90dc93d23 settled=true
{"provingJobId":"1:BASE_PARITY:993666f90dc93d23","status":"fulfilled"}
10:14:16 [10:14:16.178]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:39b436322c46de54 settled=true
{"provingJobId":"2:BASE_PARITY:39b436322c46de54","status":"fulfilled"}
10:14:16 [10:14:16.190]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwfSuID/1 with map size 1073741824
10:14:16 [10:14:16.193]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.215]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwfSuID/2 with map size 1073741824
10:14:16 [10:14:16.219]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.240]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.241]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5f8026d177971575 settled=true
{"provingJobId":"1:BASE_PARITY:5f8026d177971575","status":"fulfilled"}
10:14:16 [10:14:16.247]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:54b7228e9b2780ec settled=false
{"provingJobId":"1:BASE_PARITY:54b7228e9b2780ec","status":"pending"}
10:14:16 [10:14:16.252]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36e7457595ea0df7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:36e7457595ea0df7"}
10:14:16 [10:14:16.253]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGtcQEL/1 with map size 1073741824
10:14:16 [10:14:16.257]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.286]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.287]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:38d9526b406ef69c epochNumber=1
{"provingJobId":"1:BASE_PARITY:38d9526b406ef69c"}
10:14:16 [10:14:16.288]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:38d9526b406ef69c: Error: db error: Error: db error
10:14:16 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:38d9526b406ef69c"}
10:14:16 [10:14:16.294]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.294]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bee174438922111a epochNumber=1
{"provingJobId":"1:BASE_PARITY:bee174438922111a"}
10:14:16 [10:14:16.295]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test38e6a7/1 with map size 1073741824
10:14:16 [10:14:16.300]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.320]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bee174438922111a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bee174438922111a"}
10:14:16 [10:14:16.321]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bee174438922111a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bee174438922111a"}
10:14:16 [10:14:16.332]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.333]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35cde827768e549f epochNumber=1
{"provingJobId":"1:BASE_PARITY:35cde827768e549f"}
10:14:16 [10:14:16.334]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0zsanT/1 with map size 1073741824
10:14:16 [10:14:16.335]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.366]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:35cde827768e549f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:35cde827768e549f"}
10:14:16 [10:14:16.367]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:35cde827768e549f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:35cde827768e549f"}
10:14:16 [10:14:16.368]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:35cde827768e549f: Error: db error
10:14:16 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:35cde827768e549f"}
10:14:16 [10:14:16.380]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.382]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40a0fb6b70a18e46 epochNumber=1
{"provingJobId":"1:BASE_PARITY:40a0fb6b70a18e46"}
10:14:16 [10:14:16.384]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2elDhA/1 with map size 1073741824
10:14:16 [10:14:16.387]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.405]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:40a0fb6b70a18e46 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:40a0fb6b70a18e46"}
10:14:16 [10:14:16.405]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:40a0fb6b70a18e46 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:40a0fb6b70a18e46"}
10:14:16 [10:14:16.419]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.420]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e549bcdcd861bc5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e549bcdcd861bc5"}
10:14:16 [10:14:16.421]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMNdwg6/1 with map size 1073741824
10:14:16 [10:14:16.423]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.443]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0e549bcdcd861bc5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0e549bcdcd861bc5"}
10:14:16 [10:14:16.444]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0e549bcdcd861bc5 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0e549bcdcd861bc5"}
10:14:16 [10:14:16.445]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:0e549bcdcd861bc5 jobErr=test error: Error: db error
10:14:16 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:0e549bcdcd861bc5"}
10:14:16 [10:14:16.451]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.452]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0e34c5e7fb7c0d8d not found
{"provingJobId":"1:BASE_PARITY:0e34c5e7fb7c0d8d"}
10:14:16 [10:14:16.458]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.459]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a7bcedf70330efc2 err=test error
{"provingJoId":"1:BASE_PARITY:a7bcedf70330efc2"}
10:14:16 [10:14:16.463]
INFO:
prover-client:proving-broker Proving Broker started
10:14:16 [10:14:16.475]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2d6191852045950 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2d6191852045950"}
10:14:16 [10:14:16.478]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhtxXtU/1 with map size 1073741824
10:14:16 [10:14:16.481]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.499]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b2d6191852045950 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b2d6191852045950"}
10:14:16 [10:14:16.499]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b2d6191852045950 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b2d6191852045950"}
10:14:16 [10:14:16.515]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:2e1c6e66e26d7e67 epochNumber=2
{"provingJobId":"2:BASE_PARITY:2e1c6e66e26d7e67"}
10:14:16 [10:14:16.517]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testhtxXtU/2 with map size 1073741824
10:14:16 [10:14:16.519]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.540]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:2e1c6e66e26d7e67 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:2e1c6e66e26d7e67"}
10:14:16 [10:14:16.540]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2e1c6e66e26d7e67 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:2e1c6e66e26d7e67"}
10:14:16 [10:14:16.606]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:95bea35f701bdbc6 epochNumber=3
{"provingJobId":"3:BASE_PARITY:95bea35f701bdbc6"}
10:14:16 [10:14:16.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testhtxXtU/3 with map size 1073741824
10:14:16 [10:14:16.609]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.644]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:16 [10:14:16.645]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:14:16 [10:14:16.693]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:93fb0ea784cbac22 epochNumber=4
{"provingJobId":"4:BASE_PARITY:93fb0ea784cbac22"}
10:14:16 [10:14:16.694]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testhtxXtU/4 with map size 1073741824
10:14:16 [10:14:16.701]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.726]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:16 [10:14:16.727]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
10:14:16 [10:14:16.787]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:288e55c4ba19763f epochNumber=5
{"provingJobId":"5:BASE_PARITY:288e55c4ba19763f"}
10:14:16 [10:14:16.789]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testhtxXtU/5 with map size 1073741824
10:14:16 [10:14:16.793]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:14:16 [10:14:16.832]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:14:16 [10:14:16.832]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
10:14:16
PASS src/proving_broker/proving_broker.test.ts (
13.386 s)
10:14:16 ProvingBroker
10:14:16 Producer API
10:14:16
✓ refuses stale jobs (54 ms)
10:14:16
✓ enqueues jobs (4 ms)
10:14:16
✓ ignores duplicate jobs (4 ms)
10:14:16
✓ reports correct status when enqueuing repeat jobs (4 ms)
10:14:16
✓ throws an error in case of duplicate job IDs (32 ms)
10:14:16
✓ returns not-found status for non-existing jobs (1 ms)
10:14:16
✓ cancels jobs in queue (3 ms)
10:14:16
✓ cancels jobs in-progress (3 ms)
10:14:16
✓ returns job result if successful (13 ms)
10:14:16
✓ returns job error if failed (3 ms)
10:14:16
✓ correctly returns job status for concurrent writes (3 ms)
10:14:16
✓ refuses stale jobs (38 ms)
10:14:16
✓ enqueues jobs (38 ms)
10:14:16
✓ ignores duplicate jobs (36 ms)
10:14:16
✓ reports correct status when enqueuing repeat jobs (33 ms)
10:14:16
✓ throws an error in case of duplicate job IDs (25 ms)
10:14:16
✓ returns not-found status for non-existing jobs (2 ms)
10:14:16
✓ cancels jobs in queue (38 ms)
10:14:16
✓ cancels jobs in-progress (30 ms)
10:14:16
✓ returns job result if successful (31 ms)
10:14:16
✓ returns job error if failed (34 ms)
10:14:16
✓ correctly returns job status for concurrent writes (41 ms)
10:14:16 Consumer API
10:14:16
✓ returns undefined if no jobs are available (1 ms)
10:14:16
✓ returns jobs in priority order (2 ms)
10:14:16
✓ returns undefined if no jobs are available for the given allowList (1 ms)
10:14:16
✓ returns a job if it is in the allowList (2 ms)
10:14:16
✓ returns the most important job if it is in the allowList (2 ms)
10:14:16
✓ returns any job if filter is empty (2 ms)
10:14:16
✓ returns a new job when reporting job success (3 ms)
10:14:16
✓ returns a new job when reporting permanent error (2 ms)
10:14:16
✓ returns a new job when reporting retry-able error (2 ms)
10:14:16
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
10:14:16
✓ returns a new job if job is already in progress elsewhere (1059 ms)
10:14:16
✓ avoids sending the same job to a new agent after a restart (1007 ms)
10:14:16
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
10:14:16
✓ tracks job result if in progress (3 ms)
10:14:16
✓ tracks job result even if job is in queue (3 ms)
10:14:16
✓ ignores reported job error if unknown job (1 ms)
10:14:16
✓ ignores job result if unknown job (2 ms)
10:14:16
✓ returns undefined if no jobs are available (1 ms)
10:14:16
✓ returns jobs in priority order (47 ms)
10:14:16
✓ returns undefined if no jobs are available for the given allowList (20 ms)
10:14:16
✓ returns a job if it is in the allowList (58 ms)
10:14:16
✓ returns the most important job if it is in the allowList (64 ms)
10:14:16
✓ returns any job if filter is empty (59 ms)
10:14:16
✓ returns a new job when reporting job success (35 ms)
10:14:16
✓ returns a new job when reporting permanent error (35 ms)
10:14:16
✓ returns a new job when reporting retry-able error (39 ms)
10:14:16
✓ returns a new job when reporting progress if current one is cancelled (35 ms)
10:14:16
✓ returns a new job if job is already in progress elsewhere (1121 ms)
10:14:16
✓ avoids sending the same job to a new agent after a restart (1070 ms)
10:14:16
✓ avoids sending a completed job to a new agent after a restart (1069 ms)
10:14:16
✓ tracks job result if in progress (59 ms)
10:14:16
✓ tracks job result even if job is in queue (58 ms)
10:14:16
✓ ignores reported job error if unknown job (4 ms)
10:14:16
✓ ignores job result if unknown job (3 ms)
10:14:16 Timeouts
10:14:16
✓ tracks in progress jobs (3 ms)
10:14:16
✓ re-enqueues jobs that time out (110 ms)
10:14:16
✓ cancel stale jobs that time out (236 ms)
10:14:16
✓ keeps the jobs in progress while it is alive (204 ms)
10:14:16
✓ tracks in progress jobs (22 ms)
10:14:16
✓ re-enqueues jobs that time out (135 ms)
10:14:16
✓ cancel stale jobs that time out (291 ms)
10:14:16
✓ keeps the jobs in progress while it is alive (240 ms)
10:14:16 Retries
10:14:16
✓ retries jobs (3 ms)
10:14:16
✓ retries up to a maximum number of times (3 ms)
10:14:16
✓ passing retry=false does not retry (5 ms)
10:14:16
✓ does not retry if job is stale (30 ms)
10:14:16
✓ retries jobs (34 ms)
10:14:16
✓ retries up to a maximum number of times (47 ms)
10:14:16
✓ passing retry=false does not retry (44 ms)
10:14:16
✓ does not retry if job is stale (93 ms)
10:14:16 Database management
10:14:16
✓ re-enqueues proof requests on start (4 ms)
10:14:16
✓ restores proof results on start (1 ms)
10:14:16
✓ only re-enqueues unfinished jobs (2 ms)
10:14:16
✓ saves job when enqueued (3 ms)
10:14:16
✓ does not retain job if database fails to save (3 ms)
10:14:16
✓ saves job result (2 ms)
10:14:16
✓ saves result even if database fails to save (2 ms)
10:14:16
✓ saves job error (2 ms)
10:14:16
✓ saves job error even if database fails to save (2 ms)
10:14:16
✓ does not save job result if job is unknown (1 ms)
10:14:16
✓ does not save job error if job is unknown (1 ms)
10:14:16
✓ cleans up old jobs periodically (260 ms)
10:14:16
✓ re-enqueues proof requests on start (71 ms)
10:14:16
✓ restores proof results on start (76 ms)
10:14:16
✓ only re-enqueues unfinished jobs (64 ms)
10:14:16
✓ saves job when enqueued (32 ms)
10:14:16
✓ does not retain job if database fails to save (7 ms)
10:14:16
✓ saves job result (39 ms)
10:14:16
✓ saves result even if database fails to save (44 ms)
10:14:16
✓ saves job error (41 ms)
10:14:16
✓ saves job error even if database fails to save (30 ms)
10:14:16
✓ does not save job result if job is unknown (7 ms)
10:14:16
✓ does not save job error if job is unknown (6 ms)
10:14:16
✓ cleans up old jobs periodically (413 ms)
10:14:16
10:14:16
Test Suites: 1 passed, 1 total
10:14:16
Tests: 96 passed, 96 total
10:14:16
Snapshots: 0 total
10:14:16
Time: 13.498 s
10:14:16
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
10:14:16
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?