Parent Log:
http://ci.aztec-labs.com/392f1be801fc367c
Command: e4e37220d2336322 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/282da3bd25d49b92970e6fc85f110092e8566299
Env: REF_NAME=gh-readonly-queue/next/pr-14610-ee24aac96469b4f1cb8d70e614b2dd27bf0eb134 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 24 14:17:56 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14610_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
14:18:00 [14:18:00.621]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.628]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c1c9de384fb6346 epochNumber=42
{"provingJobId":"1:BASE_PARITY:0c1c9de384fb6346"}
14:18:00 [14:18:00.632]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:959d854a86b45bae epochNumber=1
{"provingJobId":"1:BASE_PARITY:959d854a86b45bae"}
14:18:00 [14:18:00.668]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.669]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51151f12fe71fe6e epochNumber=1
{"provingJobId":"1:BASE_PARITY:51151f12fe71fe6e"}
14:18:00 [14:18:00.670]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e7dc163357fec75 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e7dc163357fec75"}
14:18:00 [14:18:00.673]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.675]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:590e7880709af983 epochNumber=1
{"provingJobId":"1:BASE_PARITY:590e7880709af983"}
14:18:00 [14:18:00.676]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:590e7880709af983 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:590e7880709af983"}
14:18:00 [14:18:00.679]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.680]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96a7dc68a493e93a epochNumber=1
{"provingJobId":"1:BASE_PARITY:96a7dc68a493e93a"}
14:18:00 [14:18:00.682]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:96a7dc68a493e93a epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:96a7dc68a493e93a"}
14:18:00 [14:18:00.682]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:96a7dc68a493e93a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:96a7dc68a493e93a"}
14:18:00 [14:18:00.683]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:96a7dc68a493e93a epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:96a7dc68a493e93a"}
14:18:00 [14:18:00.685]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.686]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:62ea2ded91cbe188 epochNumber=1
{"provingJobId":"1:BASE_PARITY:62ea2ded91cbe188"}
14:18:00 [14:18:00.713]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.715]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.716]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3782578f4846d085 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3782578f4846d085"}
14:18:00 [14:18:00.717]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3782578f4846d085
{"provingJobId":"1:BASE_PARITY:3782578f4846d085"}
14:18:00 [14:18:00.718]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3782578f4846d085 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3782578f4846d085"}
14:18:00 [14:18:00.719]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3782578f4846d085 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3782578f4846d085"}
14:18:00 [14:18:00.721]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.722]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a38b2f6d006f8ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:9a38b2f6d006f8ff"}
14:18:00 [14:18:00.723]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9a38b2f6d006f8ff
{"provingJobId":"1:BASE_PARITY:9a38b2f6d006f8ff"}
14:18:00 [14:18:00.723]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9a38b2f6d006f8ff type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:9a38b2f6d006f8ff"}
14:18:00 [14:18:00.725]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.726]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:982cb3a9ce27cfef epochNumber=1
{"provingJobId":"1:BASE_PARITY:982cb3a9ce27cfef"}
14:18:00 [14:18:00.727]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:982cb3a9ce27cfef type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:982cb3a9ce27cfef"}
14:18:00 [14:18:00.727]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:982cb3a9ce27cfef type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:982cb3a9ce27cfef"}
14:18:00 [14:18:00.729]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.730]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09bd84572a384d0e epochNumber=1
{"provingJobId":"1:BASE_PARITY:09bd84572a384d0e"}
14:18:00 [14:18:00.730]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:09bd84572a384d0e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:09bd84572a384d0e"}
14:18:00 [14:18:00.731]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:09bd84572a384d0e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:09bd84572a384d0e"}
14:18:00 [14:18:00.733]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.735]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:382f2a624100d5d4 epochNumber=0
{"provingJobId":"1:BASE_PARITY:382f2a624100d5d4"}
14:18:00 [14:18:00.735]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:382f2a624100d5d4 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:382f2a624100d5d4"}
14:18:00 [14:18:00.736]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ef758824cf60004 epochNumber=0
{"provingJobId":"1:BASE_PARITY:1ef758824cf60004"}
14:18:00 [14:18:00.736]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:382f2a624100d5d4 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:382f2a624100d5d4"}
14:18:00 [14:18:00.736]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2bdad6378197cc4 epochNumber=0
{"provingJobId":"1:BASE_PARITY:f2bdad6378197cc4"}
14:18:00 [14:18:00.751]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.752]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.753]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ae8936e72a58769 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8ae8936e72a58769"}
14:18:00 [14:18:00.754]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:39be7f03cc6ee555 epochNumber=1
{"provingJobId":"1:BASE_PARITY:39be7f03cc6ee555"}
14:18:00 [14:18:00.755]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.756]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:871a552d8ea6d545 epochNumber=1
{"provingJobId":"1:BASE_PARITY:871a552d8ea6d545"}
14:18:00 [14:18:00.757]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.758]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d4731fb3b361529 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1d4731fb3b361529"}
14:18:00 [14:18:00.758]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7240fd44af316c5a epochNumber=1
{"provingJobId":"1:BASE_PARITY:7240fd44af316c5a"}
14:18:00 [14:18:00.758]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8723bd18562b5fbc epochNumber=2
{"provingJobId":"1:BASE_PARITY:8723bd18562b5fbc"}
14:18:00 [14:18:00.759]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb8fe2c94f2e4d66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb8fe2c94f2e4d66"}
14:18:00 [14:18:00.760]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.761]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ac408adb18f4640 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ac408adb18f4640"}
14:18:00 [14:18:00.761]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81a9eb9970846c99 epochNumber=1
{"provingJobId":"1:BASE_PARITY:81a9eb9970846c99"}
14:18:00 [14:18:00.761]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe3ee5c66e9c81e3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fe3ee5c66e9c81e3"}
14:18:00 [14:18:00.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:318657d9a92bcc79 epochNumber=1
{"provingJobId":"1:BASE_PARITY:318657d9a92bcc79"}
14:18:00 [14:18:00.763]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.764]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd4cc857ce8b1015 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd4cc857ce8b1015"}
14:18:00 [14:18:00.764]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cebe7758f2d9dce6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cebe7758f2d9dce6"}
14:18:00 [14:18:00.764]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e9c2e75ed7ac373c epochNumber=2
{"provingJobId":"1:BASE_PARITY:e9c2e75ed7ac373c"}
14:18:00 [14:18:00.765]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:495e3cc1a913f052 epochNumber=1
{"provingJobId":"1:BASE_PARITY:495e3cc1a913f052"}
14:18:00 [14:18:00.766]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.767]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba8733ac590e317f epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba8733ac590e317f"}
14:18:00 [14:18:00.767]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59b3615fb5695327 epochNumber=1
{"provingJobId":"1:BASE_PARITY:59b3615fb5695327"}
14:18:00 [14:18:00.768]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ba8733ac590e317f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ba8733ac590e317f"}
14:18:00 [14:18:00.769]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c3013f095715edf4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c3013f095715edf4"}
14:18:00 [14:18:00.770]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a9eb6ce17a116aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a9eb6ce17a116aa"}
14:18:00 [14:18:00.770]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c3013f095715edf4 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:c3013f095715edf4"}
14:18:00 [14:18:00.772]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f97b54cca925a3f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f97b54cca925a3f5"}
14:18:00 [14:18:00.773]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:031559f9aad7b004 epochNumber=1
{"provingJobId":"1:BASE_PARITY:031559f9aad7b004"}
14:18:00 [14:18:00.773]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f97b54cca925a3f5 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:f97b54cca925a3f5"}
14:18:00 [14:18:00.774]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc8c8fd3005b3857 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc8c8fd3005b3857"}
14:18:00 [14:18:00.775]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:cc8c8fd3005b3857
{"provingJobId":"1:BASE_PARITY:cc8c8fd3005b3857"}
14:18:00 [14:18:00.775]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cc8c8fd3005b3857 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:cc8c8fd3005b3857"}
14:18:00 [14:18:00.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f97f9db4c79fddc epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f97f9db4c79fddc"}
14:18:00 [14:18:00.776]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cc8c8fd3005b3857 has already been completed
{"provingJobId":"1:BASE_PARITY:cc8c8fd3005b3857"}
14:18:00 [14:18:00.777]
INFO:
prover-client:proving-broker Proving Broker started
14:18:00 [14:18:00.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:391725c24eec7a50 epochNumber=1
{"provingJobId":"1:BASE_PARITY:391725c24eec7a50"}
14:18:00 [14:18:00.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c98bfd46eca9d95e epochNumber=2
{"provingJobId":"1:BASE_PARITY:c98bfd46eca9d95e"}
14:18:01 [14:18:01.831]
INFO:
prover-client:proving-broker Proving Broker started
14:18:01 [14:18:01.832]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:391725c24eec7a50 settled=false
{"provingJobId":"1:BASE_PARITY:391725c24eec7a50","status":"pending"}
14:18:01 [14:18:01.832]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c98bfd46eca9d95e settled=false
{"provingJobId":"1:BASE_PARITY:c98bfd46eca9d95e","status":"pending"}
14:18:01 [14:18:01.833]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:391725c24eec7a50 type=BASE_PARITY startedAt=1750774680778 older agent has taken job
{"provingJobId":"1:BASE_PARITY:391725c24eec7a50"}
14:18:01 [14:18:01.834]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:391725c24eec7a50 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:391725c24eec7a50"}
14:18:01 [14:18:01.837]
INFO:
prover-client:proving-broker Proving Broker started
14:18:01 [14:18:01.838]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc75fd3e3dd01d6d epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc75fd3e3dd01d6d"}
14:18:01 [14:18:01.838]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a92f1717c0b477f epochNumber=2
{"provingJobId":"1:BASE_PARITY:7a92f1717c0b477f"}
14:18:02 [14:18:02.840]
INFO:
prover-client:proving-broker Proving Broker started
14:18:02 [14:18:02.841]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dc75fd3e3dd01d6d settled=false
{"provingJobId":"1:BASE_PARITY:dc75fd3e3dd01d6d","status":"pending"}
14:18:02 [14:18:02.841]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7a92f1717c0b477f settled=false
{"provingJobId":"1:BASE_PARITY:7a92f1717c0b477f","status":"pending"}
14:18:02 [14:18:02.842]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dc75fd3e3dd01d6d type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:dc75fd3e3dd01d6d"}
14:18:02 [14:18:02.844]
INFO:
prover-client:proving-broker Proving Broker started
14:18:02 [14:18:02.845]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e69b917b3db61fb7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e69b917b3db61fb7"}
14:18:02 [14:18:02.845]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6851f0b02e33cdc2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6851f0b02e33cdc2"}
14:18:03 [14:18:03.847]
INFO:
prover-client:proving-broker Proving Broker started
14:18:03 [14:18:03.848]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e69b917b3db61fb7 settled=false
{"provingJobId":"1:BASE_PARITY:e69b917b3db61fb7","status":"pending"}
14:18:03 [14:18:03.848]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6851f0b02e33cdc2 settled=false
{"provingJobId":"1:BASE_PARITY:6851f0b02e33cdc2","status":"pending"}
14:18:03 [14:18:03.849]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e69b917b3db61fb7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e69b917b3db61fb7"}
14:18:03 [14:18:03.849]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e69b917b3db61fb7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e69b917b3db61fb7"}
14:18:03 [14:18:03.853]
INFO:
prover-client:proving-broker Proving Broker started
14:18:03 [14:18:03.854]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fcc632972b2aa369 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fcc632972b2aa369"}
14:18:03 [14:18:03.854]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a46b5a5df413f7c4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a46b5a5df413f7c4"}
14:18:03 [14:18:03.856]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fcc632972b2aa369 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fcc632972b2aa369"}
14:18:03 [14:18:03.857]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a46b5a5df413f7c4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a46b5a5df413f7c4"}
14:18:03 [14:18:03.859]
INFO:
prover-client:proving-broker Proving Broker started
14:18:03 [14:18:03.860]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60c5c624bc160dca epochNumber=1
{"provingJobId":"1:BASE_PARITY:60c5c624bc160dca"}
14:18:03 [14:18:03.861]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7b3f6d265579ede0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7b3f6d265579ede0"}
14:18:03 [14:18:03.861]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:60c5c624bc160dca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:60c5c624bc160dca"}
14:18:03 [14:18:03.862]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:60c5c624bc160dca type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:60c5c624bc160dca"}
14:18:03 [14:18:03.863]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7b3f6d265579ede0 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7b3f6d265579ede0"}
14:18:03 [14:18:03.864]
INFO:
prover-client:proving-broker Proving Broker started
14:18:03 [14:18:03.865]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:28adc7ac3cf2136b err=test error
{"provingJoId":"1:BASE_PARITY:28adc7ac3cf2136b"}
14:18:03 [14:18:03.866]
INFO:
prover-client:proving-broker Proving Broker started
14:18:03 [14:18:03.867]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aa5fd86457a605b5 not found
{"provingJobId":"1:BASE_PARITY:aa5fd86457a605b5"}
14:18:03 [14:18:03.869]
INFO:
prover-client:proving-broker Proving Broker started
14:18:03 [14:18:03.869]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab70d0cb9096dd78 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab70d0cb9096dd78"}
14:18:03 [14:18:03.870]
INFO:
prover-client:proving-broker Proving Broker started
14:18:03 [14:18:03.871]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2349be8eba79da67 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2349be8eba79da67"}
14:18:03 [14:18:03.973]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2349be8eba79da67 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2349be8eba79da67"}
14:18:03 [14:18:03.980]
INFO:
prover-client:proving-broker Proving Broker started
14:18:03 [14:18:03.981]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4cac09d7f7eeffc epochNumber=1
{"provingJobId":"1:BASE_PARITY:e4cac09d7f7eeffc"}
14:18:04 [14:18:04.083]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e4cac09d7f7eeffc timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:e4cac09d7f7eeffc"}
14:18:04 [14:18:04.090]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0f36cba4e126b175 epochNumber=10
{"provingJobId":"1:BASE_PARITY:0f36cba4e126b175"}
14:18:04 [14:18:04.110]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:04 [14:18:04.219]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f96a7edb3fe40cf2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f96a7edb3fe40cf2"}
14:18:04 [14:18:04.400]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f96a7edb3fe40cf2 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f96a7edb3fe40cf2"}
14:18:04 [14:18:04.434]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.435]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cab10b5125c99094 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cab10b5125c99094"}
14:18:04 [14:18:04.437]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cab10b5125c99094 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:cab10b5125c99094"}
14:18:04 [14:18:04.438]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ec29ceec4f6e1a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ec29ceec4f6e1a4"}
14:18:04 [14:18:04.441]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8ec29ceec4f6e1a4 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:8ec29ceec4f6e1a4"}
14:18:04 [14:18:04.442]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8ec29ceec4f6e1a4 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:8ec29ceec4f6e1a4"}
14:18:04 [14:18:04.444]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.444]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e2415642ed00d24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e2415642ed00d24"}
14:18:04 [14:18:04.445]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3e2415642ed00d24 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3e2415642ed00d24"}
14:18:04 [14:18:04.446]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c200a631db6d790 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c200a631db6d790"}
14:18:04 [14:18:04.448]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4c200a631db6d790 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:4c200a631db6d790"}
14:18:04 [14:18:04.448]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f676f34e932f13a epochNumber=3
{"provingJobId":"1:BASE_PARITY:3f676f34e932f13a"}
14:18:04 [14:18:04.473]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:04 [14:18:04.475]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4c200a631db6d790 err=test error
{"provingJoId":"1:BASE_PARITY:4c200a631db6d790"}
14:18:04 [14:18:04.476]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.477]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:02df1ca9e09facc2 settled=false
{"provingJobId":"1:BASE_PARITY:02df1ca9e09facc2","status":"pending"}
14:18:04 [14:18:04.477]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:66104b990dda0b22 settled=false
{"provingJobId":"1:BASE_PARITY:66104b990dda0b22","status":"pending"}
14:18:04 [14:18:04.479]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.479]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7e440a92f7b7016e settled=true
{"provingJobId":"1:BASE_PARITY:7e440a92f7b7016e","status":"fulfilled"}
14:18:04 [14:18:04.479]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:f1c8e2d6741b10e6 settled=true
{"provingJobId":"2:BASE_PARITY:f1c8e2d6741b10e6","status":"fulfilled"}
14:18:04 [14:18:04.480]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.481]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9637c175cddc480f settled=true
{"provingJobId":"1:BASE_PARITY:9637c175cddc480f","status":"fulfilled"}
14:18:04 [14:18:04.481]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:17e7f3833cef550c settled=false
{"provingJobId":"1:BASE_PARITY:17e7f3833cef550c","status":"pending"}
14:18:04 [14:18:04.482]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.483]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b56a420f516c5617 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b56a420f516c5617"}
14:18:04 [14:18:04.486]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.487]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd0d5dccfa8d201b epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd0d5dccfa8d201b"}
14:18:04 [14:18:04.488]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:bd0d5dccfa8d201b: Error: db error: Error: db error
14:18:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:bd0d5dccfa8d201b"}
14:18:04 [14:18:04.490]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.491]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:69e1c0b66ab7a78e epochNumber=1
{"provingJobId":"1:BASE_PARITY:69e1c0b66ab7a78e"}
14:18:04 [14:18:04.491]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:69e1c0b66ab7a78e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:69e1c0b66ab7a78e"}
14:18:04 [14:18:04.492]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:69e1c0b66ab7a78e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:69e1c0b66ab7a78e"}
14:18:04 [14:18:04.493]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.494]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c09b8ccecad70aaf epochNumber=1
{"provingJobId":"1:BASE_PARITY:c09b8ccecad70aaf"}
14:18:04 [14:18:04.494]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c09b8ccecad70aaf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c09b8ccecad70aaf"}
14:18:04 [14:18:04.495]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c09b8ccecad70aaf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c09b8ccecad70aaf"}
14:18:04 [14:18:04.496]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:c09b8ccecad70aaf: Error: db error
14:18:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:c09b8ccecad70aaf"}
14:18:04 [14:18:04.498]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.498]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35d47b8347dc657e epochNumber=1
{"provingJobId":"1:BASE_PARITY:35d47b8347dc657e"}
14:18:04 [14:18:04.498]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:35d47b8347dc657e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:35d47b8347dc657e"}
14:18:04 [14:18:04.499]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:35d47b8347dc657e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:35d47b8347dc657e"}
14:18:04 [14:18:04.500]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.501]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:226cce618091aae5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:226cce618091aae5"}
14:18:04 [14:18:04.501]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:226cce618091aae5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:226cce618091aae5"}
14:18:04 [14:18:04.501]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:226cce618091aae5 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:226cce618091aae5"}
14:18:04 [14:18:04.502]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:226cce618091aae5 jobErr=test error: Error: db error
14:18:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:226cce618091aae5"}
14:18:04 [14:18:04.503]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.503]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4fb157a5a02017e0 not found
{"provingJobId":"1:BASE_PARITY:4fb157a5a02017e0"}
14:18:04 [14:18:04.505]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.505]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6acab70cbf19b5f5 err=test error
{"provingJoId":"1:BASE_PARITY:6acab70cbf19b5f5"}
14:18:04 [14:18:04.506]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.517]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4474d0d756146bac epochNumber=1
{"provingJobId":"1:BASE_PARITY:4474d0d756146bac"}
14:18:04 [14:18:04.518]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4474d0d756146bac type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4474d0d756146bac"}
14:18:04 [14:18:04.519]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4474d0d756146bac type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4474d0d756146bac"}
14:18:04 [14:18:04.529]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:00ecb29b2fdb94a6 epochNumber=2
{"provingJobId":"2:BASE_PARITY:00ecb29b2fdb94a6"}
14:18:04 [14:18:04.530]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:00ecb29b2fdb94a6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:00ecb29b2fdb94a6"}
14:18:04 [14:18:04.530]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:00ecb29b2fdb94a6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:00ecb29b2fdb94a6"}
14:18:04 [14:18:04.591]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:cf6147eef20c6014 epochNumber=3
{"provingJobId":"3:BASE_PARITY:cf6147eef20c6014"}
14:18:04 [14:18:04.608]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:04 [14:18:04.653]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:7d9c23bd078c03d7 epochNumber=4
{"provingJobId":"4:BASE_PARITY:7d9c23bd078c03d7"}
14:18:04 [14:18:04.658]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:04 [14:18:04.717]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:0d42f4ba130614d1 epochNumber=5
{"provingJobId":"5:BASE_PARITY:0d42f4ba130614d1"}
14:18:04 [14:18:04.735]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:04 [14:18:04.771]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81947e9597de74c8 epochNumber=42
{"provingJobId":"1:BASE_PARITY:81947e9597de74c8"}
14:18:04 [14:18:04.773]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testzwgBUH/42 with map size 1073741824
14:18:04 [14:18:04.777]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:04 [14:18:04.808]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:6c4210bd407a668b epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c4210bd407a668b"}
14:18:04 [14:18:04.814]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.815]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4b284fc37515754 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c4b284fc37515754"}
14:18:04 [14:18:04.816]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkx0sG0/1 with map size 1073741824
14:18:04 [14:18:04.822]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:04 [14:18:04.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ffd5aa2355de3d4c epochNumber=1
{"provingJobId":"1:BASE_PARITY:ffd5aa2355de3d4c"}
14:18:04 [14:18:04.856]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.857]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6685431889cf930e epochNumber=1
{"provingJobId":"1:BASE_PARITY:6685431889cf930e"}
14:18:04 [14:18:04.857]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test44gyVd/1 with map size 1073741824
14:18:04 [14:18:04.861]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:04 [14:18:04.883]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6685431889cf930e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:6685431889cf930e"}
14:18:04 [14:18:04.888]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.889]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87bcb07608786e1b epochNumber=1
{"provingJobId":"1:BASE_PARITY:87bcb07608786e1b"}
14:18:04 [14:18:04.890]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXLbN43/1 with map size 1073741824
14:18:04 [14:18:04.894]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:04 [14:18:04.917]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:87bcb07608786e1b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:87bcb07608786e1b"}
14:18:04 [14:18:04.918]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:87bcb07608786e1b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:87bcb07608786e1b"}
14:18:04 [14:18:04.929]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:87bcb07608786e1b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:87bcb07608786e1b"}
14:18:04 [14:18:04.938]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85075d783f490266 epochNumber=1
{"provingJobId":"1:BASE_PARITY:85075d783f490266"}
14:18:04 [14:18:04.941]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1f0u8n/1 with map size 1073741824
14:18:04 [14:18:04.944]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:04 [14:18:04.968]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.971]
INFO:
prover-client:proving-broker Proving Broker started
14:18:04 [14:18:04.972]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e371a73ee88345cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:e371a73ee88345cb"}
14:18:04 [14:18:04.972]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFU7vB7/1 with map size 1073741824
14:18:04 [14:18:04.974]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:04 [14:18:04.990]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e371a73ee88345cb
{"provingJobId":"1:BASE_PARITY:e371a73ee88345cb"}
14:18:04 [14:18:04.991]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e371a73ee88345cb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e371a73ee88345cb"}
14:18:04 [14:18:04.991]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e371a73ee88345cb type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:e371a73ee88345cb"}
14:18:05 [14:18:05.001]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.002]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7d7bb6310fc94386 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7d7bb6310fc94386"}
14:18:05 [14:18:05.003]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testw4qQAc/1 with map size 1073741824
14:18:05 [14:18:05.006]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.029]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7d7bb6310fc94386
{"provingJobId":"1:BASE_PARITY:7d7bb6310fc94386"}
14:18:05 [14:18:05.030]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7d7bb6310fc94386 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7d7bb6310fc94386"}
14:18:05 [14:18:05.044]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.046]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:848de34fcab88597 epochNumber=1
{"provingJobId":"1:BASE_PARITY:848de34fcab88597"}
14:18:05 [14:18:05.050]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoPuA5C/1 with map size 1073741824
14:18:05 [14:18:05.054]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.073]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:848de34fcab88597 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:848de34fcab88597"}
14:18:05 [14:18:05.073]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:848de34fcab88597 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:848de34fcab88597"}
14:18:05 [14:18:05.082]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:429fc16bc751cb85 epochNumber=1
{"provingJobId":"1:BASE_PARITY:429fc16bc751cb85"}
14:18:05 [14:18:05.083]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test174hBh/1 with map size 1073741824
14:18:05 [14:18:05.086]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.102]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:429fc16bc751cb85 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:429fc16bc751cb85"}
14:18:05 [14:18:05.102]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:429fc16bc751cb85 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:429fc16bc751cb85"}
14:18:05 [14:18:05.110]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.110]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c26dd29b929914d epochNumber=0
{"provingJobId":"1:BASE_PARITY:4c26dd29b929914d"}
14:18:05 [14:18:05.112]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testQ8jyPI/0 with map size 1073741824
14:18:05 [14:18:05.117]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.137]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4c26dd29b929914d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4c26dd29b929914d"}
14:18:05 [14:18:05.138]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d8396d202efe55a1 epochNumber=0
{"provingJobId":"1:BASE_PARITY:d8396d202efe55a1"}
14:18:05 [14:18:05.138]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:4c26dd29b929914d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:4c26dd29b929914d"}
14:18:05 [14:18:05.138]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab6c1cc8e7b7e2e2 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ab6c1cc8e7b7e2e2"}
14:18:05 [14:18:05.152]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.154]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.155]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4b0ae871712e154 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e4b0ae871712e154"}
14:18:05 [14:18:05.155]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqZEc2y/2 with map size 1073741824
14:18:05 [14:18:05.157]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.173]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cfba1a16c28a70ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:cfba1a16c28a70ec"}
14:18:05 [14:18:05.174]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqZEc2y/1 with map size 1073741824
14:18:05 [14:18:05.177]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.195]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.195]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f376e6aac0f07f52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f376e6aac0f07f52"}
14:18:05 [14:18:05.196]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWH9BBd/1 with map size 1073741824
14:18:05 [14:18:05.200]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.221]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec2f8a53fe9bac0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec2f8a53fe9bac0b"}
14:18:05 [14:18:05.223]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqlYwC7/1 with map size 1073741824
14:18:05 [14:18:05.225]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.245]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7f6ba247b5095280 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7f6ba247b5095280"}
14:18:05 [14:18:05.254]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25cf0f65ced7d201 epochNumber=2
{"provingJobId":"1:BASE_PARITY:25cf0f65ced7d201"}
14:18:05 [14:18:05.255]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqlYwC7/2 with map size 1073741824
14:18:05 [14:18:05.257]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.276]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb3a636783ceb6fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb3a636783ceb6fd"}
14:18:05 [14:18:05.288]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.289]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b77cb59a1214807 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b77cb59a1214807"}
14:18:05 [14:18:05.289]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCX1TWI/1 with map size 1073741824
14:18:05 [14:18:05.291]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.313]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1915dde67950541b epochNumber=1
{"provingJobId":"1:BASE_PARITY:1915dde67950541b"}
14:18:05 [14:18:05.321]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2b2f10c75cf601d epochNumber=2
{"provingJobId":"1:BASE_PARITY:c2b2f10c75cf601d"}
14:18:05 [14:18:05.323]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCX1TWI/2 with map size 1073741824
14:18:05 [14:18:05.327]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:92ae0020705e3692 epochNumber=1
{"provingJobId":"1:BASE_PARITY:92ae0020705e3692"}
14:18:05 [14:18:05.356]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70e3ba52fcee03cc epochNumber=1
{"provingJobId":"1:BASE_PARITY:70e3ba52fcee03cc"}
14:18:05 [14:18:05.357]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaivxeK/1 with map size 1073741824
14:18:05 [14:18:05.359]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.375]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05450d0f5a556cdc epochNumber=1
{"provingJobId":"1:BASE_PARITY:05450d0f5a556cdc"}
14:18:05 [14:18:05.381]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ec91ce584593577 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5ec91ce584593577"}
14:18:05 [14:18:05.381]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testaivxeK/2 with map size 1073741824
14:18:05 [14:18:05.383]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.398]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba05457230874f4d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba05457230874f4d"}
14:18:05 [14:18:05.406]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.406]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c184f61bf930c26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c184f61bf930c26"}
14:18:05 [14:18:05.408]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJhgdvO/1 with map size 1073741824
14:18:05 [14:18:05.411]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.427]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f120a85dab94e352 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f120a85dab94e352"}
14:18:05 [14:18:05.433]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0c184f61bf930c26 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0c184f61bf930c26"}
14:18:05 [14:18:05.446]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dad1eea40af16a36 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dad1eea40af16a36"}
14:18:05 [14:18:05.449]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWm88RG/1 with map size 1073741824
14:18:05 [14:18:05.452]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.481]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec9ff77dabd8d71e epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec9ff77dabd8d71e"}
14:18:05 [14:18:05.488]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dad1eea40af16a36 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:dad1eea40af16a36"}
14:18:05 [14:18:05.500]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.501]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe34d2bf2c91796f epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe34d2bf2c91796f"}
14:18:05 [14:18:05.501]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLbqQBY/1 with map size 1073741824
14:18:05 [14:18:05.503]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.522]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af01d0a126a4ba63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af01d0a126a4ba63"}
14:18:05 [14:18:05.527]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fe34d2bf2c91796f type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:fe34d2bf2c91796f"}
14:18:05 [14:18:05.531]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.532]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c34a4e8b915276a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c34a4e8b915276a7"}
14:18:05 [14:18:05.534]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbkwEwZ/1 with map size 1073741824
14:18:05 [14:18:05.538]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.556]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c34a4e8b915276a7
{"provingJobId":"1:BASE_PARITY:c34a4e8b915276a7"}
14:18:05 [14:18:05.557]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c34a4e8b915276a7 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c34a4e8b915276a7"}
14:18:05 [14:18:05.564]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc9af6660c80456c epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc9af6660c80456c"}
14:18:05 [14:18:05.573]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c34a4e8b915276a7 has already been completed
{"provingJobId":"1:BASE_PARITY:c34a4e8b915276a7"}
14:18:05 [14:18:05.577]
INFO:
prover-client:proving-broker Proving Broker started
14:18:05 [14:18:05.578]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5b5076a26ac61ce epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5b5076a26ac61ce"}
14:18:05 [14:18:05.579]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfh6ytv/1 with map size 1073741824
14:18:05 [14:18:05.580]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:05 [14:18:05.598]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c8be27327e4e5690 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c8be27327e4e5690"}
14:18:05 [14:18:05.599]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfh6ytv/2 with map size 1073741824
14:18:05 [14:18:05.601]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:06 [14:18:06.668]
INFO:
prover-client:proving-broker Proving Broker started
14:18:06 [14:18:06.674]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e5b5076a26ac61ce settled=false
{"provingJobId":"1:BASE_PARITY:e5b5076a26ac61ce","status":"pending"}
14:18:06 [14:18:06.675]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c8be27327e4e5690 settled=false
{"provingJobId":"1:BASE_PARITY:c8be27327e4e5690","status":"pending"}
14:18:06 [14:18:06.676]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e5b5076a26ac61ce type=BASE_PARITY startedAt=1750774685617 older agent has taken job
{"provingJobId":"1:BASE_PARITY:e5b5076a26ac61ce"}
14:18:06 [14:18:06.677]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e5b5076a26ac61ce type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:e5b5076a26ac61ce"}
14:18:06 [14:18:06.680]
INFO:
prover-client:proving-broker Proving Broker started
14:18:06 [14:18:06.681]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aab14215a878dece epochNumber=1
{"provingJobId":"1:BASE_PARITY:aab14215a878dece"}
14:18:06 [14:18:06.681]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPNZnvk/1 with map size 1073741824
14:18:06 [14:18:06.683]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:06 [14:18:06.701]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:caa302a6d0fcb669 epochNumber=2
{"provingJobId":"1:BASE_PARITY:caa302a6d0fcb669"}
14:18:06 [14:18:06.702]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testPNZnvk/2 with map size 1073741824
14:18:06 [14:18:06.703]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:07 [14:18:07.721]
INFO:
prover-client:proving-broker Proving Broker started
14:18:07 [14:18:07.723]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:aab14215a878dece settled=false
{"provingJobId":"1:BASE_PARITY:aab14215a878dece","status":"pending"}
14:18:07 [14:18:07.724]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:caa302a6d0fcb669 settled=false
{"provingJobId":"1:BASE_PARITY:caa302a6d0fcb669","status":"pending"}
14:18:07 [14:18:07.725]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aab14215a878dece type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:aab14215a878dece"}
14:18:07 [14:18:07.728]
INFO:
prover-client:proving-broker Proving Broker started
14:18:07 [14:18:07.729]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:797d985c9bd2628e epochNumber=1
{"provingJobId":"1:BASE_PARITY:797d985c9bd2628e"}
14:18:07 [14:18:07.730]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjgLoyA/1 with map size 1073741824
14:18:07 [14:18:07.731]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:07 [14:18:07.750]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6dd5b9e9ad577107 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6dd5b9e9ad577107"}
14:18:07 [14:18:07.751]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjgLoyA/2 with map size 1073741824
14:18:07 [14:18:07.753]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:08 [14:18:08.770]
INFO:
prover-client:proving-broker Proving Broker started
14:18:08 [14:18:08.773]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:797d985c9bd2628e settled=false
{"provingJobId":"1:BASE_PARITY:797d985c9bd2628e","status":"pending"}
14:18:08 [14:18:08.775]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6dd5b9e9ad577107 settled=false
{"provingJobId":"1:BASE_PARITY:6dd5b9e9ad577107","status":"pending"}
14:18:08 [14:18:08.777]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:797d985c9bd2628e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:797d985c9bd2628e"}
14:18:08 [14:18:08.777]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:797d985c9bd2628e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:797d985c9bd2628e"}
14:18:08 [14:18:08.787]
INFO:
prover-client:proving-broker Proving Broker started
14:18:08 [14:18:08.788]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53ba0834818b23f4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53ba0834818b23f4"}
14:18:08 [14:18:08.789]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFV2Gf4/1 with map size 1073741824
14:18:08 [14:18:08.791]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:08 [14:18:08.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00763dcb418212ec epochNumber=2
{"provingJobId":"1:BASE_PARITY:00763dcb418212ec"}
14:18:08 [14:18:08.809]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFV2Gf4/2 with map size 1073741824
14:18:08 [14:18:08.811]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:08 [14:18:08.837]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:53ba0834818b23f4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:53ba0834818b23f4"}
14:18:08 [14:18:08.844]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:00763dcb418212ec type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:00763dcb418212ec"}
14:18:08 [14:18:08.856]
INFO:
prover-client:proving-broker Proving Broker started
14:18:08 [14:18:08.857]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4da603ff1ea48991 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4da603ff1ea48991"}
14:18:08 [14:18:08.858]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiinIy8/1 with map size 1073741824
14:18:08 [14:18:08.861]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:08 [14:18:08.884]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09fc53831270fa4d epochNumber=2
{"provingJobId":"1:BASE_PARITY:09fc53831270fa4d"}
14:18:08 [14:18:08.885]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testiinIy8/2 with map size 1073741824
14:18:08 [14:18:08.891]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:08 [14:18:08.911]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4da603ff1ea48991 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4da603ff1ea48991"}
14:18:08 [14:18:08.912]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4da603ff1ea48991 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4da603ff1ea48991"}
14:18:08 [14:18:08.939]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:09fc53831270fa4d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:09fc53831270fa4d"}
14:18:08 [14:18:08.952]
INFO:
prover-client:proving-broker Proving Broker started
14:18:08 [14:18:08.953]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c6547adefff869f6 err=test error
{"provingJoId":"1:BASE_PARITY:c6547adefff869f6"}
14:18:08 [14:18:08.958]
INFO:
prover-client:proving-broker Proving Broker started
14:18:08 [14:18:08.958]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a1ac43c37ab0d62d not found
{"provingJobId":"1:BASE_PARITY:a1ac43c37ab0d62d"}
14:18:08 [14:18:08.961]
INFO:
prover-client:proving-broker Proving Broker started
14:18:08 [14:18:08.962]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35a91cbba6c54a38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:35a91cbba6c54a38"}
14:18:08 [14:18:08.962]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6AmwJI/1 with map size 1073741824
14:18:08 [14:18:08.965]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:08 [14:18:08.986]
INFO:
prover-client:proving-broker Proving Broker started
14:18:08 [14:18:08.987]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2d0aa2a6883d6a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2d0aa2a6883d6a0"}
14:18:08 [14:18:08.988]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test081ICO/1 with map size 1073741824
14:18:08 [14:18:08.993]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.115]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b2d0aa2a6883d6a0 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b2d0aa2a6883d6a0"}
14:18:09 [14:18:09.129]
INFO:
prover-client:proving-broker Proving Broker started
14:18:09 [14:18:09.130]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70304bc3a3cdba19 epochNumber=1
{"provingJobId":"1:BASE_PARITY:70304bc3a3cdba19"}
14:18:09 [14:18:09.135]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJVWxHv/1 with map size 1073741824
14:18:09 [14:18:09.137]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.282]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:70304bc3a3cdba19 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:70304bc3a3cdba19"}
14:18:09 [14:18:09.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7bbdf9b7136c980d epochNumber=10
{"provingJobId":"1:BASE_PARITY:7bbdf9b7136c980d"}
14:18:09 [14:18:09.287]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testJVWxHv/10 with map size 1073741824
14:18:09 [14:18:09.290]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.309]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:09 [14:18:09.310]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:18:09 [14:18:09.436]
INFO:
prover-client:proving-broker Proving Broker started
14:18:09 [14:18:09.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0512ef81989a5ea1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0512ef81989a5ea1"}
14:18:09 [14:18:09.438]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsgiERS/1 with map size 1073741824
14:18:09 [14:18:09.440]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.615]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0512ef81989a5ea1 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:0512ef81989a5ea1"}
14:18:09 [14:18:09.668]
INFO:
prover-client:proving-broker Proving Broker started
14:18:09 [14:18:09.669]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a1becc1d0e1b936 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a1becc1d0e1b936"}
14:18:09 [14:18:09.669]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFftc7T/1 with map size 1073741824
14:18:09 [14:18:09.672]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.691]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:5a1becc1d0e1b936 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:5a1becc1d0e1b936"}
14:18:09 [14:18:09.699]
INFO:
prover-client:proving-broker Proving Broker started
14:18:09 [14:18:09.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f9b6501fb172e047 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f9b6501fb172e047"}
14:18:09 [14:18:09.701]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test23HaZb/1 with map size 1073741824
14:18:09 [14:18:09.703]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.722]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f9b6501fb172e047 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f9b6501fb172e047"}
14:18:09 [14:18:09.723]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f9b6501fb172e047 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:f9b6501fb172e047"}
14:18:09 [14:18:09.733]
INFO:
prover-client:proving-broker Proving Broker started
14:18:09 [14:18:09.735]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08441c23767abf72 epochNumber=1
{"provingJobId":"1:BASE_PARITY:08441c23767abf72"}
14:18:09 [14:18:09.736]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrTSggk/1 with map size 1073741824
14:18:09 [14:18:09.738]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.756]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:08441c23767abf72 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:08441c23767abf72"}
14:18:09 [14:18:09.766]
INFO:
prover-client:proving-broker Proving Broker started
14:18:09 [14:18:09.767]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97ad38ab690c62b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:97ad38ab690c62b8"}
14:18:09 [14:18:09.768]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxXz52s/1 with map size 1073741824
14:18:09 [14:18:09.771]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.789]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:97ad38ab690c62b8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:97ad38ab690c62b8"}
14:18:09 [14:18:09.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae74b2f584b6808d epochNumber=3
{"provingJobId":"1:BASE_PARITY:ae74b2f584b6808d"}
14:18:09 [14:18:09.792]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testxXz52s/3 with map size 1073741824
14:18:09 [14:18:09.795]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.844]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:09 [14:18:09.845]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:18:09 [14:18:09.846]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:97ad38ab690c62b8 err=test error
{"provingJoId":"1:BASE_PARITY:97ad38ab690c62b8"}
14:18:09 [14:18:09.853]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvmfo4s/1 with map size 1073741824
14:18:09 [14:18:09.856]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.873]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvmfo4s/2 with map size 1073741824
14:18:09 [14:18:09.876]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.892]
INFO:
prover-client:proving-broker Proving Broker started
14:18:09 [14:18:09.894]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6af9005d60b4dd6e settled=false
{"provingJobId":"1:BASE_PARITY:6af9005d60b4dd6e","status":"pending"}
14:18:09 [14:18:09.895]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:51d147e819503370 settled=false
{"provingJobId":"1:BASE_PARITY:51d147e819503370","status":"pending"}
14:18:09 [14:18:09.907]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testY3YmzF/1 with map size 1073741824
14:18:09 [14:18:09.909]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.933]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testY3YmzF/2 with map size 1073741824
14:18:09 [14:18:09.936]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:09 [14:18:09.968]
INFO:
prover-client:proving-broker Proving Broker started
14:18:09 [14:18:09.971]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8703ce9e625f9259 settled=true
{"provingJobId":"1:BASE_PARITY:8703ce9e625f9259","status":"fulfilled"}
14:18:09 [14:18:09.973]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:4bde43f2d730bb96 settled=true
{"provingJobId":"2:BASE_PARITY:4bde43f2d730bb96","status":"fulfilled"}
14:18:09 [14:18:09.981]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNrI5m7/1 with map size 1073741824
14:18:09 [14:18:09.984]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.008]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNrI5m7/2 with map size 1073741824
14:18:10 [14:18:10.012]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.041]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.043]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a0aa6eb98f417059 settled=true
{"provingJobId":"1:BASE_PARITY:a0aa6eb98f417059","status":"fulfilled"}
14:18:10 [14:18:10.046]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2699f8caefe805c9 settled=false
{"provingJobId":"1:BASE_PARITY:2699f8caefe805c9","status":"pending"}
14:18:10 [14:18:10.051]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.052]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:114d38ed8f3b7b74 epochNumber=1
{"provingJobId":"1:BASE_PARITY:114d38ed8f3b7b74"}
14:18:10 [14:18:10.054]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhrKX9r/1 with map size 1073741824
14:18:10 [14:18:10.057]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.078]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fe536e663ed5ad5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fe536e663ed5ad5"}
14:18:10 [14:18:10.080]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:4fe536e663ed5ad5: Error: db error: Error: db error
14:18:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:4fe536e663ed5ad5"}
14:18:10 [14:18:10.085]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.085]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82710d3bb76101c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:82710d3bb76101c3"}
14:18:10 [14:18:10.086]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuLNLOe/1 with map size 1073741824
14:18:10 [14:18:10.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.106]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:82710d3bb76101c3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:82710d3bb76101c3"}
14:18:10 [14:18:10.106]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:82710d3bb76101c3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:82710d3bb76101c3"}
14:18:10 [14:18:10.122]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.123]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66d8c3915f81713e epochNumber=1
{"provingJobId":"1:BASE_PARITY:66d8c3915f81713e"}
14:18:10 [14:18:10.127]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkEXM1p/1 with map size 1073741824
14:18:10 [14:18:10.131]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.153]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:66d8c3915f81713e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:66d8c3915f81713e"}
14:18:10 [14:18:10.153]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:66d8c3915f81713e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:66d8c3915f81713e"}
14:18:10 [14:18:10.154]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:66d8c3915f81713e: Error: db error
14:18:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:66d8c3915f81713e"}
14:18:10 [14:18:10.158]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6118527ff96eca8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d6118527ff96eca8"}
14:18:10 [14:18:10.159]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoDWd7D/1 with map size 1073741824
14:18:10 [14:18:10.161]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.179]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d6118527ff96eca8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d6118527ff96eca8"}
14:18:10 [14:18:10.180]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d6118527ff96eca8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d6118527ff96eca8"}
14:18:10 [14:18:10.191]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.192]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aae9b101189231c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aae9b101189231c8"}
14:18:10 [14:18:10.193]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgUK4Or/1 with map size 1073741824
14:18:10 [14:18:10.195]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.218]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aae9b101189231c8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:aae9b101189231c8"}
14:18:10 [14:18:10.218]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:aae9b101189231c8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:aae9b101189231c8"}
14:18:10 [14:18:10.219]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:aae9b101189231c8 jobErr=test error: Error: db error
14:18:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:aae9b101189231c8"}
14:18:10 [14:18:10.234]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.235]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:22ec8e78c2155677 not found
{"provingJobId":"1:BASE_PARITY:22ec8e78c2155677"}
14:18:10 [14:18:10.241]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.242]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:59a97f05bc6e54b6 err=test error
{"provingJoId":"1:BASE_PARITY:59a97f05bc6e54b6"}
14:18:10 [14:18:10.245]
INFO:
prover-client:proving-broker Proving Broker started
14:18:10 [14:18:10.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:923bf86d3d5ac9a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:923bf86d3d5ac9a8"}
14:18:10 [14:18:10.257]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAYCHf6/1 with map size 1073741824
14:18:10 [14:18:10.261]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.280]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:923bf86d3d5ac9a8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:923bf86d3d5ac9a8"}
14:18:10 [14:18:10.281]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:923bf86d3d5ac9a8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:923bf86d3d5ac9a8"}
14:18:10 [14:18:10.300]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:1cf8e1f8b40f2730 epochNumber=2
{"provingJobId":"2:BASE_PARITY:1cf8e1f8b40f2730"}
14:18:10 [14:18:10.302]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAYCHf6/2 with map size 1073741824
14:18:10 [14:18:10.305]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.326]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:1cf8e1f8b40f2730 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:1cf8e1f8b40f2730"}
14:18:10 [14:18:10.327]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:1cf8e1f8b40f2730 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:1cf8e1f8b40f2730"}
14:18:10 [14:18:10.396]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:584b7041a8c2c042 epochNumber=3
{"provingJobId":"3:BASE_PARITY:584b7041a8c2c042"}
14:18:10 [14:18:10.398]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testAYCHf6/3 with map size 1073741824
14:18:10 [14:18:10.400]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.430]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:10 [14:18:10.431]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
14:18:10 [14:18:10.489]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:13b5649793cb2587 epochNumber=4
{"provingJobId":"4:BASE_PARITY:13b5649793cb2587"}
14:18:10 [14:18:10.490]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testAYCHf6/4 with map size 1073741824
14:18:10 [14:18:10.493]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.511]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:10 [14:18:10.512]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
14:18:10 [14:18:10.571]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:51654c737c052366 epochNumber=5
{"provingJobId":"5:BASE_PARITY:51654c737c052366"}
14:18:10 [14:18:10.573]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testAYCHf6/5 with map size 1073741824
14:18:10 [14:18:10.575]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
14:18:10 [14:18:10.599]
INFO:
prover-client:proving-broker Cleaned up jobs=1
14:18:10 [14:18:10.600]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
14:18:10
PASS src/proving_broker/proving_broker.test.ts (
13.337 s)
14:18:10 ProvingBroker
14:18:10 Producer API
14:18:10
✓ refuses stale jobs (50 ms)
14:18:10
✓ enqueues jobs (5 ms)
14:18:10
✓ ignores duplicate jobs (5 ms)
14:18:10
✓ reports correct status when enqueuing repeat jobs (5 ms)
14:18:10
✓ throws an error in case of duplicate job IDs (27 ms)
14:18:10
✓ returns not-found status for non-existing jobs (2 ms)
14:18:10
✓ cancels jobs in queue (6 ms)
14:18:10
✓ cancels jobs in-progress (4 ms)
14:18:10
✓ returns job result if successful (4 ms)
14:18:10
✓ returns job error if failed (4 ms)
14:18:10
✓ correctly returns job status for concurrent writes (16 ms)
14:18:10
✓ refuses stale jobs (42 ms)
14:18:10
✓ enqueues jobs (42 ms)
14:18:10
✓ ignores duplicate jobs (30 ms)
14:18:10
✓ reports correct status when enqueuing repeat jobs (47 ms)
14:18:10
✓ throws an error in case of duplicate job IDs (32 ms)
14:18:10
✓ returns not-found status for non-existing jobs (3 ms)
14:18:10
✓ cancels jobs in queue (29 ms)
14:18:10
✓ cancels jobs in-progress (41 ms)
14:18:10
✓ returns job result if successful (40 ms)
14:18:10
✓ returns job error if failed (28 ms)
14:18:10
✓ correctly returns job status for concurrent writes (42 ms)
14:18:10 Consumer API
14:18:10
✓ returns undefined if no jobs are available (2 ms)
14:18:10
✓ returns jobs in priority order (3 ms)
14:18:10
✓ returns undefined if no jobs are available for the given allowList (1 ms)
14:18:10
✓ returns a job if it is in the allowList (2 ms)
14:18:10
✓ returns the most important job if it is in the allowList (2 ms)
14:18:10
✓ returns any job if filter is empty (2 ms)
14:18:10
✓ returns a new job when reporting job success (2 ms)
14:18:10
✓ returns a new job when reporting permanent error (2 ms)
14:18:10
✓ returns a new job when reporting retry-able error (2 ms)
14:18:10
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
14:18:10
✓ returns a new job if job is already in progress elsewhere (1057 ms)
14:18:10
✓ avoids sending the same job to a new agent after a restart (1006 ms)
14:18:10
✓ avoids sending a completed job to a new agent after a restart (1006 ms)
14:18:10
✓ tracks job result if in progress (6 ms)
14:18:10
✓ tracks job result even if job is in queue (5 ms)
14:18:10
✓ ignores reported job error if unknown job (2 ms)
14:18:10
✓ ignores job result if unknown job (2 ms)
14:18:10
✓ returns undefined if no jobs are available (2 ms)
14:18:10
✓ returns jobs in priority order (40 ms)
14:18:10
✓ returns undefined if no jobs are available for the given allowList (23 ms)
14:18:10
✓ returns a job if it is in the allowList (68 ms)
14:18:10
✓ returns the most important job if it is in the allowList (68 ms)
14:18:10
✓ returns any job if filter is empty (49 ms)
14:18:10
✓ returns a new job when reporting job success (38 ms)
14:18:10
✓ returns a new job when reporting permanent error (56 ms)
14:18:10
✓ returns a new job when reporting retry-able error (31 ms)
14:18:10
✓ returns a new job when reporting progress if current one is cancelled (46 ms)
14:18:10
✓ returns a new job if job is already in progress elsewhere (1103 ms)
14:18:10
✓ avoids sending the same job to a new agent after a restart (1048 ms)
14:18:10
✓ avoids sending a completed job to a new agent after a restart (1058 ms)
14:18:10
✓ tracks job result if in progress (69 ms)
14:18:10
✓ tracks job result even if job is in queue (96 ms)
14:18:10
✓ ignores reported job error if unknown job (3 ms)
14:18:10
✓ ignores job result if unknown job (4 ms)
14:18:10 Timeouts
14:18:10
✓ tracks in progress jobs (2 ms)
14:18:10
✓ re-enqueues jobs that time out (109 ms)
14:18:10
✓ cancel stale jobs that time out (239 ms)
14:18:10
✓ keeps the jobs in progress while it is alive (213 ms)
14:18:10
✓ tracks in progress jobs (24 ms)
14:18:10
✓ re-enqueues jobs that time out (142 ms)
14:18:10
✓ cancel stale jobs that time out (307 ms)
14:18:10
✓ keeps the jobs in progress while it is alive (230 ms)
14:18:10 Retries
14:18:10
✓ retries jobs (5 ms)
14:18:10
✓ retries up to a maximum number of times (5 ms)
14:18:10
✓ passing retry=false does not retry (3 ms)
14:18:10
✓ does not retry if job is stale (29 ms)
14:18:10
✓ retries jobs (31 ms)
14:18:10
✓ retries up to a maximum number of times (35 ms)
14:18:10
✓ passing retry=false does not retry (32 ms)
14:18:10
✓ does not retry if job is stale (84 ms)
14:18:10 Database management
14:18:10
✓ re-enqueues proof requests on start (2 ms)
14:18:10
✓ restores proof results on start (1 ms)
14:18:10
✓ only re-enqueues unfinished jobs (1 ms)
14:18:10
✓ saves job when enqueued (3 ms)
14:18:10
✓ does not retain job if database fails to save (3 ms)
14:18:10
✓ saves job result (2 ms)
14:18:10
✓ saves result even if database fails to save (4 ms)
14:18:10
✓ saves job error (3 ms)
14:18:10
✓ saves job error even if database fails to save (2 ms)
14:18:10
✓ does not save job result if job is unknown (2 ms)
14:18:10
✓ does not save job error if job is unknown (2 ms)
14:18:10
✓ cleans up old jobs periodically (262 ms)
14:18:10
✓ re-enqueues proof requests on start (54 ms)
14:18:10
✓ restores proof results on start (74 ms)
14:18:10
✓ only re-enqueues unfinished jobs (71 ms)
14:18:10
✓ saves job when enqueued (26 ms)
14:18:10
✓ does not retain job if database fails to save (5 ms)
14:18:10
✓ saves job result (37 ms)
14:18:10
✓ saves result even if database fails to save (36 ms)
14:18:10
✓ saves job error (33 ms)
14:18:10
✓ saves job error even if database fails to save (38 ms)
14:18:10
✓ does not save job result if job is unknown (9 ms)
14:18:10
✓ does not save job error if job is unknown (6 ms)
14:18:10
✓ cleans up old jobs periodically (408 ms)
14:18:10
14:18:10
Test Suites: 1 passed, 1 total
14:18:10
Tests: 96 passed, 96 total
14:18:10
Snapshots: 0 total
14:18:10
Time: 13.418 s
14:18:10
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
14:18:10
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?