Parent Log:
http://ci.aztec-labs.com/1142fda5c44bde41
Command: 53eb328a1446f762 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/a48851a8c6884f4b9615a154f8862a549722d25e
Env: REF_NAME=gh-readonly-queue/next/pr-14680-3b5835195491b53670250790b18c14d922718574 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Tue Jun 17 11:29:03 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14680_amd64_x4-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
11:29:08 [11:29:08.671]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.679]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e3e096519ab8f15 epochNumber=42
{"provingJobId":"1:BASE_PARITY:7e3e096519ab8f15"}
11:29:08 [11:29:08.682]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:9d1c5b6a10e459fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d1c5b6a10e459fd"}
11:29:08 [11:29:08.721]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.722]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea137c3fa9a47185 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea137c3fa9a47185"}
11:29:08 [11:29:08.723]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c72036f0d4cd2d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c72036f0d4cd2d0"}
11:29:08 [11:29:08.725]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.726]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08df713ea0600f59 epochNumber=1
{"provingJobId":"1:BASE_PARITY:08df713ea0600f59"}
11:29:08 [11:29:08.727]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:08df713ea0600f59 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:08df713ea0600f59"}
11:29:08 [11:29:08.729]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.730]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35ba881d12b75b32 epochNumber=1
{"provingJobId":"1:BASE_PARITY:35ba881d12b75b32"}
11:29:08 [11:29:08.731]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:35ba881d12b75b32 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:35ba881d12b75b32"}
11:29:08 [11:29:08.732]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:35ba881d12b75b32 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:35ba881d12b75b32"}
11:29:08 [11:29:08.732]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:35ba881d12b75b32 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:35ba881d12b75b32"}
11:29:08 [11:29:08.734]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.734]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c7f16e3d4b91732 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c7f16e3d4b91732"}
11:29:08 [11:29:08.762]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.764]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.765]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ece2928b99f8e40 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ece2928b99f8e40"}
11:29:08 [11:29:08.766]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9ece2928b99f8e40
{"provingJobId":"1:BASE_PARITY:9ece2928b99f8e40"}
11:29:08 [11:29:08.766]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9ece2928b99f8e40 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9ece2928b99f8e40"}
11:29:08 [11:29:08.766]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9ece2928b99f8e40 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:9ece2928b99f8e40"}
11:29:08 [11:29:08.768]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.769]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b103e0098459022 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b103e0098459022"}
11:29:08 [11:29:08.770]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6b103e0098459022
{"provingJobId":"1:BASE_PARITY:6b103e0098459022"}
11:29:08 [11:29:08.770]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6b103e0098459022 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:6b103e0098459022"}
11:29:08 [11:29:08.772]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.773]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da936c78291b0a24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:da936c78291b0a24"}
11:29:08 [11:29:08.773]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:da936c78291b0a24 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:da936c78291b0a24"}
11:29:08 [11:29:08.773]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:da936c78291b0a24 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:da936c78291b0a24"}
11:29:08 [11:29:08.775]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:939a99cf595ae2d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:939a99cf595ae2d3"}
11:29:08 [11:29:08.776]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:939a99cf595ae2d3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:939a99cf595ae2d3"}
11:29:08 [11:29:08.776]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:939a99cf595ae2d3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:939a99cf595ae2d3"}
11:29:08 [11:29:08.778]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b20ac220c418305c epochNumber=0
{"provingJobId":"1:BASE_PARITY:b20ac220c418305c"}
11:29:08 [11:29:08.778]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b20ac220c418305c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b20ac220c418305c"}
11:29:08 [11:29:08.779]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2557c9f5366503f4 epochNumber=0
{"provingJobId":"1:BASE_PARITY:2557c9f5366503f4"}
11:29:08 [11:29:08.779]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b20ac220c418305c epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b20ac220c418305c"}
11:29:08 [11:29:08.779]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57c916f08bc7c9e0 epochNumber=0
{"provingJobId":"1:BASE_PARITY:57c916f08bc7c9e0"}
11:29:08 [11:29:08.781]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.782]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1adde0661eb9347e epochNumber=2
{"provingJobId":"1:BASE_PARITY:1adde0661eb9347e"}
11:29:08 [11:29:08.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc63a0899dc26c37 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc63a0899dc26c37"}
11:29:08 [11:29:08.784]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.784]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4861e8522d53c177 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4861e8522d53c177"}
11:29:08 [11:29:08.786]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1089c54a3a044490 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1089c54a3a044490"}
11:29:08 [11:29:08.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3203ee87641a30a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3203ee87641a30a"}
11:29:08 [11:29:08.787]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b054d09e501e3ae7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b054d09e501e3ae7"}
11:29:08 [11:29:08.787]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a87e30f17d5b784d epochNumber=1
{"provingJobId":"1:BASE_PARITY:a87e30f17d5b784d"}
11:29:08 [11:29:08.788]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.788]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7fa21ef6ee652e1a epochNumber=1
{"provingJobId":"1:BASE_PARITY:7fa21ef6ee652e1a"}
11:29:08 [11:29:08.789]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5411d1f3e13b7b04 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5411d1f3e13b7b04"}
11:29:08 [11:29:08.789]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f6f8b6b5e46852a epochNumber=2
{"provingJobId":"1:BASE_PARITY:4f6f8b6b5e46852a"}
11:29:08 [11:29:08.789]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7470d9dc319e940d epochNumber=1
{"provingJobId":"1:BASE_PARITY:7470d9dc319e940d"}
11:29:08 [11:29:08.790]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0f4d08e10d6cede epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0f4d08e10d6cede"}
11:29:08 [11:29:08.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66c48aa1e084bbb8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66c48aa1e084bbb8"}
11:29:08 [11:29:08.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:343143f059416d50 epochNumber=2
{"provingJobId":"1:BASE_PARITY:343143f059416d50"}
11:29:08 [11:29:08.792]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:522b39ce542f9764 epochNumber=1
{"provingJobId":"1:BASE_PARITY:522b39ce542f9764"}
11:29:08 [11:29:08.793]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.793]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec19feeaaad5768b epochNumber=1
{"provingJobId":"1:BASE_PARITY:ec19feeaaad5768b"}
11:29:08 [11:29:08.793]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17dc2005e21bc294 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17dc2005e21bc294"}
11:29:08 [11:29:08.794]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ec19feeaaad5768b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ec19feeaaad5768b"}
11:29:08 [11:29:08.795]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.795]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ec7ab6a6e7d8003 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ec7ab6a6e7d8003"}
11:29:08 [11:29:08.796]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de4ddade12bb97fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:de4ddade12bb97fb"}
11:29:08 [11:29:08.796]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2ec7ab6a6e7d8003 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:2ec7ab6a6e7d8003"}
11:29:08 [11:29:08.797]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.797]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e2c8b92f4d26169 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2e2c8b92f4d26169"}
11:29:08 [11:29:08.798]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1b39800cb47f7c4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1b39800cb47f7c4"}
11:29:08 [11:29:08.798]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2e2c8b92f4d26169 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:2e2c8b92f4d26169"}
11:29:08 [11:29:08.799]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c03d15a7692b1a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c03d15a7692b1a1"}
11:29:08 [11:29:08.800]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:8c03d15a7692b1a1
{"provingJobId":"1:BASE_PARITY:8c03d15a7692b1a1"}
11:29:08 [11:29:08.805]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8c03d15a7692b1a1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:8c03d15a7692b1a1"}
11:29:08 [11:29:08.806]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc77294ffc2dcf17 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bc77294ffc2dcf17"}
11:29:08 [11:29:08.807]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8c03d15a7692b1a1 has already been completed
{"provingJobId":"1:BASE_PARITY:8c03d15a7692b1a1"}
11:29:08 [11:29:08.808]
INFO:
prover-client:proving-broker Proving Broker started
11:29:08 [11:29:08.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3fc5d7ec6367efb6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3fc5d7ec6367efb6"}
11:29:08 [11:29:08.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a651ded51732413 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7a651ded51732413"}
11:29:09 [11:29:09.860]
INFO:
prover-client:proving-broker Proving Broker started
11:29:09 [11:29:09.861]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3fc5d7ec6367efb6 settled=false
{"provingJobId":"1:BASE_PARITY:3fc5d7ec6367efb6","status":"pending"}
11:29:09 [11:29:09.861]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7a651ded51732413 settled=false
{"provingJobId":"1:BASE_PARITY:7a651ded51732413","status":"pending"}
11:29:09 [11:29:09.862]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3fc5d7ec6367efb6 type=BASE_PARITY startedAt=1750159748808 older agent has taken job
{"provingJobId":"1:BASE_PARITY:3fc5d7ec6367efb6"}
11:29:09 [11:29:09.862]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3fc5d7ec6367efb6 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:3fc5d7ec6367efb6"}
11:29:09 [11:29:09.863]
INFO:
prover-client:proving-broker Proving Broker started
11:29:09 [11:29:09.864]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42e7aff86b1776eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:42e7aff86b1776eb"}
11:29:09 [11:29:09.864]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32a9830688703262 epochNumber=2
{"provingJobId":"1:BASE_PARITY:32a9830688703262"}
11:29:10 [11:29:10.864]
INFO:
prover-client:proving-broker Proving Broker started
11:29:10 [11:29:10.865]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:42e7aff86b1776eb settled=false
{"provingJobId":"1:BASE_PARITY:42e7aff86b1776eb","status":"pending"}
11:29:10 [11:29:10.866]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:32a9830688703262 settled=false
{"provingJobId":"1:BASE_PARITY:32a9830688703262","status":"pending"}
11:29:10 [11:29:10.868]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:42e7aff86b1776eb type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:42e7aff86b1776eb"}
11:29:10 [11:29:10.871]
INFO:
prover-client:proving-broker Proving Broker started
11:29:10 [11:29:10.871]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8d3926bcc78fbf7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f8d3926bcc78fbf7"}
11:29:10 [11:29:10.872]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:551b33edf2322691 epochNumber=2
{"provingJobId":"1:BASE_PARITY:551b33edf2322691"}
11:29:11 [11:29:11.873]
INFO:
prover-client:proving-broker Proving Broker started
11:29:11 [11:29:11.874]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f8d3926bcc78fbf7 settled=false
{"provingJobId":"1:BASE_PARITY:f8d3926bcc78fbf7","status":"pending"}
11:29:11 [11:29:11.875]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:551b33edf2322691 settled=false
{"provingJobId":"1:BASE_PARITY:551b33edf2322691","status":"pending"}
11:29:11 [11:29:11.875]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f8d3926bcc78fbf7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f8d3926bcc78fbf7"}
11:29:11 [11:29:11.876]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f8d3926bcc78fbf7 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f8d3926bcc78fbf7"}
11:29:11 [11:29:11.877]
INFO:
prover-client:proving-broker Proving Broker started
11:29:11 [11:29:11.878]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:10874d96426f56a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:10874d96426f56a1"}
11:29:11 [11:29:11.878]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2911d0177d6e30b8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2911d0177d6e30b8"}
11:29:11 [11:29:11.879]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:10874d96426f56a1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:10874d96426f56a1"}
11:29:11 [11:29:11.879]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2911d0177d6e30b8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2911d0177d6e30b8"}
11:29:11 [11:29:11.881]
INFO:
prover-client:proving-broker Proving Broker started
11:29:11 [11:29:11.881]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c7314e3c81befec epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c7314e3c81befec"}
11:29:11 [11:29:11.882]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df050b6ae52e6834 epochNumber=2
{"provingJobId":"1:BASE_PARITY:df050b6ae52e6834"}
11:29:11 [11:29:11.882]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6c7314e3c81befec type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6c7314e3c81befec"}
11:29:11 [11:29:11.882]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6c7314e3c81befec type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6c7314e3c81befec"}
11:29:11 [11:29:11.882]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:df050b6ae52e6834 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:df050b6ae52e6834"}
11:29:11 [11:29:11.883]
INFO:
prover-client:proving-broker Proving Broker started
11:29:11 [11:29:11.884]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:5cd199e9e2ff5053 err=test error
{"provingJoId":"1:BASE_PARITY:5cd199e9e2ff5053"}
11:29:11 [11:29:11.885]
INFO:
prover-client:proving-broker Proving Broker started
11:29:11 [11:29:11.886]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:afaa8e20b8afcae7 not found
{"provingJobId":"1:BASE_PARITY:afaa8e20b8afcae7"}
11:29:11 [11:29:11.887]
INFO:
prover-client:proving-broker Proving Broker started
11:29:11 [11:29:11.888]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd672fc8f0d2a020 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cd672fc8f0d2a020"}
11:29:11 [11:29:11.889]
INFO:
prover-client:proving-broker Proving Broker started
11:29:11 [11:29:11.889]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b51451162f3a9c95 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b51451162f3a9c95"}
11:29:11 [11:29:11.990]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b51451162f3a9c95 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b51451162f3a9c95"}
11:29:11 [11:29:11.997]
INFO:
prover-client:proving-broker Proving Broker started
11:29:11 [11:29:11.998]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f6550a5222d67885 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f6550a5222d67885"}
11:29:12 [11:29:12.099]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f6550a5222d67885 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f6550a5222d67885"}
11:29:12 [11:29:12.107]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4cfeabd15a7aa3e9 epochNumber=10
{"provingJobId":"1:BASE_PARITY:4cfeabd15a7aa3e9"}
11:29:12 [11:29:12.125]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:12 [11:29:12.234]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:901a3bac02b9e0ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:901a3bac02b9e0ca"}
11:29:12 [11:29:12.389]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:901a3bac02b9e0ca timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:901a3bac02b9e0ca"}
11:29:12 [11:29:12.438]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.439]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1431501c0df47853 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1431501c0df47853"}
11:29:12 [11:29:12.445]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1431501c0df47853 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:1431501c0df47853"}
11:29:12 [11:29:12.447]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.447]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:545c5094756d323a epochNumber=1
{"provingJobId":"1:BASE_PARITY:545c5094756d323a"}
11:29:12 [11:29:12.453]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:545c5094756d323a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:545c5094756d323a"}
11:29:12 [11:29:12.454]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:545c5094756d323a type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:545c5094756d323a"}
11:29:12 [11:29:12.455]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.455]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af55ab84c61bcf4f epochNumber=1
{"provingJobId":"1:BASE_PARITY:af55ab84c61bcf4f"}
11:29:12 [11:29:12.456]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:af55ab84c61bcf4f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:af55ab84c61bcf4f"}
11:29:12 [11:29:12.458]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.458]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b85c4b2388b15d54 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b85c4b2388b15d54"}
11:29:12 [11:29:12.459]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b85c4b2388b15d54 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b85c4b2388b15d54"}
11:29:12 [11:29:12.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7d0e06c25439687e epochNumber=3
{"provingJobId":"1:BASE_PARITY:7d0e06c25439687e"}
11:29:12 [11:29:12.484]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:12 [11:29:12.485]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b85c4b2388b15d54 err=test error
{"provingJoId":"1:BASE_PARITY:b85c4b2388b15d54"}
11:29:12 [11:29:12.486]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.487]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d354d4fe92759a3d settled=false
{"provingJobId":"1:BASE_PARITY:d354d4fe92759a3d","status":"pending"}
11:29:12 [11:29:12.487]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d5f56994b3de1748 settled=false
{"provingJobId":"1:BASE_PARITY:d5f56994b3de1748","status":"pending"}
11:29:12 [11:29:12.489]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.489]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fc07d53164385e1c settled=true
{"provingJobId":"1:BASE_PARITY:fc07d53164385e1c","status":"fulfilled"}
11:29:12 [11:29:12.490]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:316d692a8423595c settled=true
{"provingJobId":"2:BASE_PARITY:316d692a8423595c","status":"fulfilled"}
11:29:12 [11:29:12.491]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.491]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:702a5384a6dfa43b settled=true
{"provingJobId":"1:BASE_PARITY:702a5384a6dfa43b","status":"fulfilled"}
11:29:12 [11:29:12.491]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ca87db03c4692e36 settled=false
{"provingJobId":"1:BASE_PARITY:ca87db03c4692e36","status":"pending"}
11:29:12 [11:29:12.492]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.494]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26871d92721adf25 epochNumber=1
{"provingJobId":"1:BASE_PARITY:26871d92721adf25"}
11:29:12 [11:29:12.496]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.497]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:30dec2835e183cb5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:30dec2835e183cb5"}
11:29:12 [11:29:12.498]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:30dec2835e183cb5: Error: db error: Error: db error
11:29:12 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:30dec2835e183cb5"}
11:29:12 [11:29:12.500]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52a112fa1ec7b161 epochNumber=1
{"provingJobId":"1:BASE_PARITY:52a112fa1ec7b161"}
11:29:12 [11:29:12.501]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:52a112fa1ec7b161 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:52a112fa1ec7b161"}
11:29:12 [11:29:12.501]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:52a112fa1ec7b161 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:52a112fa1ec7b161"}
11:29:12 [11:29:12.502]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.503]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bff2f6b0de956467 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bff2f6b0de956467"}
11:29:12 [11:29:12.503]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bff2f6b0de956467 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bff2f6b0de956467"}
11:29:12 [11:29:12.503]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bff2f6b0de956467 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bff2f6b0de956467"}
11:29:12 [11:29:12.504]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:bff2f6b0de956467: Error: db error
11:29:12 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:bff2f6b0de956467"}
11:29:12 [11:29:12.505]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f818d6dd0063d69e epochNumber=1
{"provingJobId":"1:BASE_PARITY:f818d6dd0063d69e"}
11:29:12 [11:29:12.506]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f818d6dd0063d69e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f818d6dd0063d69e"}
11:29:12 [11:29:12.506]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f818d6dd0063d69e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f818d6dd0063d69e"}
11:29:12 [11:29:12.507]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.508]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f25a2faeedeb4032 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f25a2faeedeb4032"}
11:29:12 [11:29:12.508]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f25a2faeedeb4032 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f25a2faeedeb4032"}
11:29:12 [11:29:12.508]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f25a2faeedeb4032 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f25a2faeedeb4032"}
11:29:12 [11:29:12.509]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:f25a2faeedeb4032 jobErr=test error: Error: db error
11:29:12 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:f25a2faeedeb4032"}
11:29:12 [11:29:12.510]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.511]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:23d6c74aa76b1e32 not found
{"provingJobId":"1:BASE_PARITY:23d6c74aa76b1e32"}
11:29:12 [11:29:12.512]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.513]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4c2e67843e1d24ba err=test error
{"provingJoId":"1:BASE_PARITY:4c2e67843e1d24ba"}
11:29:12 [11:29:12.514]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.524]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1f118681b063493 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c1f118681b063493"}
11:29:12 [11:29:12.525]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c1f118681b063493 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c1f118681b063493"}
11:29:12 [11:29:12.525]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c1f118681b063493 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c1f118681b063493"}
11:29:12 [11:29:12.535]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:94952b147b60acd8 epochNumber=2
{"provingJobId":"2:BASE_PARITY:94952b147b60acd8"}
11:29:12 [11:29:12.536]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:94952b147b60acd8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:94952b147b60acd8"}
11:29:12 [11:29:12.537]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:94952b147b60acd8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:94952b147b60acd8"}
11:29:12 [11:29:12.598]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:d7bd0814f59ebc85 epochNumber=3
{"provingJobId":"3:BASE_PARITY:d7bd0814f59ebc85"}
11:29:12 [11:29:12.615]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:12 [11:29:12.660]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:5e56ca2dde93d160 epochNumber=4
{"provingJobId":"4:BASE_PARITY:5e56ca2dde93d160"}
11:29:12 [11:29:12.668]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:12 [11:29:12.721]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:17bb58ed2803d356 epochNumber=5
{"provingJobId":"5:BASE_PARITY:17bb58ed2803d356"}
11:29:12 [11:29:12.745]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:12 [11:29:12.777]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c03bd91fa5a8b27 epochNumber=42
{"provingJobId":"1:BASE_PARITY:2c03bd91fa5a8b27"}
11:29:12 [11:29:12.779]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testmNHHew/42 with map size 1073741824
11:29:12 [11:29:12.786]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:12 [11:29:12.811]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:aedd6411c8d02095 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aedd6411c8d02095"}
11:29:12 [11:29:12.817]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.817]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b65c3a19b03880e epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b65c3a19b03880e"}
11:29:12 [11:29:12.818]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testw9ZAqB/1 with map size 1073741824
11:29:12 [11:29:12.820]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:12 [11:29:12.842]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:afd20c972ed28995 epochNumber=1
{"provingJobId":"1:BASE_PARITY:afd20c972ed28995"}
11:29:12 [11:29:12.856]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.856]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53736ddc2030fd47 epochNumber=1
{"provingJobId":"1:BASE_PARITY:53736ddc2030fd47"}
11:29:12 [11:29:12.857]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhhq2Xo/1 with map size 1073741824
11:29:12 [11:29:12.860]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:12 [11:29:12.879]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:53736ddc2030fd47 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:53736ddc2030fd47"}
11:29:12 [11:29:12.883]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e779ef21bf4a78f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e779ef21bf4a78f0"}
11:29:12 [11:29:12.884]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjCdVFV/1 with map size 1073741824
11:29:12 [11:29:12.885]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:12 [11:29:12.903]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e779ef21bf4a78f0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e779ef21bf4a78f0"}
11:29:12 [11:29:12.903]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e779ef21bf4a78f0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e779ef21bf4a78f0"}
11:29:12 [11:29:12.909]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e779ef21bf4a78f0 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e779ef21bf4a78f0"}
11:29:12 [11:29:12.913]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.914]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b248087bc01ec32 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b248087bc01ec32"}
11:29:12 [11:29:12.914]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-tests7M29F/1 with map size 1073741824
11:29:12 [11:29:12.916]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:12 [11:29:12.939]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.941]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.942]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbe2fdd5bed7b79c epochNumber=1
{"provingJobId":"1:BASE_PARITY:dbe2fdd5bed7b79c"}
11:29:12 [11:29:12.943]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVau89R/1 with map size 1073741824
11:29:12 [11:29:12.944]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:12 [11:29:12.965]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:dbe2fdd5bed7b79c
{"provingJobId":"1:BASE_PARITY:dbe2fdd5bed7b79c"}
11:29:12 [11:29:12.965]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dbe2fdd5bed7b79c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dbe2fdd5bed7b79c"}
11:29:12 [11:29:12.965]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dbe2fdd5bed7b79c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:dbe2fdd5bed7b79c"}
11:29:12 [11:29:12.976]
INFO:
prover-client:proving-broker Proving Broker started
11:29:12 [11:29:12.977]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a2a13b8e948bffa epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a2a13b8e948bffa"}
11:29:12 [11:29:12.978]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhq5dYg/1 with map size 1073741824
11:29:12 [11:29:12.984]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:12 [11:29:12.999]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3a2a13b8e948bffa
{"provingJobId":"1:BASE_PARITY:3a2a13b8e948bffa"}
11:29:12 [11:29:12.999]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3a2a13b8e948bffa type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3a2a13b8e948bffa"}
11:29:13 [11:29:13.006]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.007]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a08dc0077370f8b2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a08dc0077370f8b2"}
11:29:13 [11:29:13.008]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYp4SUX/1 with map size 1073741824
11:29:13 [11:29:13.010]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.020]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a08dc0077370f8b2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a08dc0077370f8b2"}
11:29:13 [11:29:13.020]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a08dc0077370f8b2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a08dc0077370f8b2"}
11:29:13 [11:29:13.026]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.027]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dcf6e39be51f90d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dcf6e39be51f90d1"}
11:29:13 [11:29:13.028]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testl7KhyX/1 with map size 1073741824
11:29:13 [11:29:13.030]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.052]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dcf6e39be51f90d1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:dcf6e39be51f90d1"}
11:29:13 [11:29:13.053]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dcf6e39be51f90d1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:dcf6e39be51f90d1"}
11:29:13 [11:29:13.059]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.060]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8de1ad7d400560f epochNumber=0
{"provingJobId":"1:BASE_PARITY:e8de1ad7d400560f"}
11:29:13 [11:29:13.061]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testKZGD8I/0 with map size 1073741824
11:29:13 [11:29:13.063]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.073]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e8de1ad7d400560f epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e8de1ad7d400560f"}
11:29:13 [11:29:13.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78ad1e9c7d971507 epochNumber=0
{"provingJobId":"1:BASE_PARITY:78ad1e9c7d971507"}
11:29:13 [11:29:13.074]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e8de1ad7d400560f epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e8de1ad7d400560f"}
11:29:13 [11:29:13.074]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26ba6bbd4332d991 epochNumber=0
{"provingJobId":"1:BASE_PARITY:26ba6bbd4332d991"}
11:29:13 [11:29:13.086]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.088]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.089]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44970ea750bd722b epochNumber=2
{"provingJobId":"1:BASE_PARITY:44970ea750bd722b"}
11:29:13 [11:29:13.090]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnCB3u2/2 with map size 1073741824
11:29:13 [11:29:13.092]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.103]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:18bff31eadcc8945 epochNumber=1
{"provingJobId":"1:BASE_PARITY:18bff31eadcc8945"}
11:29:13 [11:29:13.104]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnCB3u2/1 with map size 1073741824
11:29:13 [11:29:13.106]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.118]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:974e79fb6e2fbf1a epochNumber=1
{"provingJobId":"1:BASE_PARITY:974e79fb6e2fbf1a"}
11:29:13 [11:29:13.119]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKe7r8Y/1 with map size 1073741824
11:29:13 [11:29:13.121]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.138]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.139]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:39a64f84e8b4c0b0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:39a64f84e8b4c0b0"}
11:29:13 [11:29:13.142]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8Ic04U/1 with map size 1073741824
11:29:13 [11:29:13.147]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.164]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae80cf003ef3fd3b epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae80cf003ef3fd3b"}
11:29:13 [11:29:13.170]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5d551e4c16a0b3d epochNumber=2
{"provingJobId":"1:BASE_PARITY:f5d551e4c16a0b3d"}
11:29:13 [11:29:13.171]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8Ic04U/2 with map size 1073741824
11:29:13 [11:29:13.173]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.189]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8cfc3d4033185d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f8cfc3d4033185d5"}
11:29:13 [11:29:13.199]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3abfdeda744a79d8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3abfdeda744a79d8"}
11:29:13 [11:29:13.201]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWL1nmk/1 with map size 1073741824
11:29:13 [11:29:13.204]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.220]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9e372e57a4bcddf epochNumber=1
{"provingJobId":"1:BASE_PARITY:a9e372e57a4bcddf"}
11:29:13 [11:29:13.226]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9370dada05921a68 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9370dada05921a68"}
11:29:13 [11:29:13.227]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWL1nmk/2 with map size 1073741824
11:29:13 [11:29:13.230]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.252]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b61afea12092e7e epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b61afea12092e7e"}
11:29:13 [11:29:13.265]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c182517a457172ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:c182517a457172ac"}
11:29:13 [11:29:13.267]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-teste1nv6T/1 with map size 1073741824
11:29:13 [11:29:13.269]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:811f0728e165cefb epochNumber=1
{"provingJobId":"1:BASE_PARITY:811f0728e165cefb"}
11:29:13 [11:29:13.292]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a010ce05eb8ba362 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a010ce05eb8ba362"}
11:29:13 [11:29:13.294]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-teste1nv6T/2 with map size 1073741824
11:29:13 [11:29:13.299]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.317]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c398d94e288876b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c398d94e288876b6"}
11:29:13 [11:29:13.327]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.328]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:185a81810891a988 epochNumber=1
{"provingJobId":"1:BASE_PARITY:185a81810891a988"}
11:29:13 [11:29:13.329]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwgdg2o/1 with map size 1073741824
11:29:13 [11:29:13.332]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.350]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1182c4ed9412a144 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1182c4ed9412a144"}
11:29:13 [11:29:13.355]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:185a81810891a988 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:185a81810891a988"}
11:29:13 [11:29:13.364]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.365]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13dbcfa1ed0e1503 epochNumber=1
{"provingJobId":"1:BASE_PARITY:13dbcfa1ed0e1503"}
11:29:13 [11:29:13.365]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5xMpX4/1 with map size 1073741824
11:29:13 [11:29:13.367]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.384]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8cd10079ca14c0d8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8cd10079ca14c0d8"}
11:29:13 [11:29:13.390]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:13dbcfa1ed0e1503 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:13dbcfa1ed0e1503"}
11:29:13 [11:29:13.400]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.401]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9bc3631c8281eced epochNumber=1
{"provingJobId":"1:BASE_PARITY:9bc3631c8281eced"}
11:29:13 [11:29:13.401]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8n041U/1 with map size 1073741824
11:29:13 [11:29:13.403]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.421]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6124cff78501c14c epochNumber=1
{"provingJobId":"1:BASE_PARITY:6124cff78501c14c"}
11:29:13 [11:29:13.428]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9bc3631c8281eced type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:9bc3631c8281eced"}
11:29:13 [11:29:13.435]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.436]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5715150270e63596 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5715150270e63596"}
11:29:13 [11:29:13.436]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvXeqvN/1 with map size 1073741824
11:29:13 [11:29:13.440]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.458]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5715150270e63596
{"provingJobId":"1:BASE_PARITY:5715150270e63596"}
11:29:13 [11:29:13.459]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5715150270e63596 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5715150270e63596"}
11:29:13 [11:29:13.468]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ecd49cd7030a0a24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ecd49cd7030a0a24"}
11:29:13 [11:29:13.476]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5715150270e63596 has already been completed
{"provingJobId":"1:BASE_PARITY:5715150270e63596"}
11:29:13 [11:29:13.480]
INFO:
prover-client:proving-broker Proving Broker started
11:29:13 [11:29:13.480]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a68c252ebc40921c epochNumber=1
{"provingJobId":"1:BASE_PARITY:a68c252ebc40921c"}
11:29:13 [11:29:13.481]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkeMopg/1 with map size 1073741824
11:29:13 [11:29:13.482]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:13 [11:29:13.504]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44c434d5411fea49 epochNumber=2
{"provingJobId":"1:BASE_PARITY:44c434d5411fea49"}
11:29:13 [11:29:13.505]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkeMopg/2 with map size 1073741824
11:29:13 [11:29:13.508]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:14 [11:29:14.573]
INFO:
prover-client:proving-broker Proving Broker started
11:29:14 [11:29:14.581]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a68c252ebc40921c settled=false
{"provingJobId":"1:BASE_PARITY:a68c252ebc40921c","status":"pending"}
11:29:14 [11:29:14.583]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:44c434d5411fea49 settled=false
{"provingJobId":"1:BASE_PARITY:44c434d5411fea49","status":"pending"}
11:29:14 [11:29:14.584]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a68c252ebc40921c type=BASE_PARITY startedAt=1750159753521 older agent has taken job
{"provingJobId":"1:BASE_PARITY:a68c252ebc40921c"}
11:29:14 [11:29:14.585]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a68c252ebc40921c type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:a68c252ebc40921c"}
11:29:14 [11:29:14.589]
INFO:
prover-client:proving-broker Proving Broker started
11:29:14 [11:29:14.590]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5fab00360d524fa5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5fab00360d524fa5"}
11:29:14 [11:29:14.591]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdCszMS/1 with map size 1073741824
11:29:14 [11:29:14.593]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:14 [11:29:14.613]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9ee67f870ec565d epochNumber=2
{"provingJobId":"1:BASE_PARITY:b9ee67f870ec565d"}
11:29:14 [11:29:14.614]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testdCszMS/2 with map size 1073741824
11:29:14 [11:29:14.616]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:15 [11:29:15.634]
INFO:
prover-client:proving-broker Proving Broker started
11:29:15 [11:29:15.641]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5fab00360d524fa5 settled=false
{"provingJobId":"1:BASE_PARITY:5fab00360d524fa5","status":"pending"}
11:29:15 [11:29:15.648]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b9ee67f870ec565d settled=false
{"provingJobId":"1:BASE_PARITY:b9ee67f870ec565d","status":"pending"}
11:29:15 [11:29:15.649]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5fab00360d524fa5 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:5fab00360d524fa5"}
11:29:15 [11:29:15.657]
INFO:
prover-client:proving-broker Proving Broker started
11:29:15 [11:29:15.658]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d694ff5e30274e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d694ff5e30274e5"}
11:29:15 [11:29:15.659]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZWXUAI/1 with map size 1073741824
11:29:15 [11:29:15.661]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:15 [11:29:15.678]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5feb6feb5fdb147 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f5feb6feb5fdb147"}
11:29:15 [11:29:15.679]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZWXUAI/2 with map size 1073741824
11:29:15 [11:29:15.681]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:16 [11:29:16.700]
INFO:
prover-client:proving-broker Proving Broker started
11:29:16 [11:29:16.702]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0d694ff5e30274e5 settled=false
{"provingJobId":"1:BASE_PARITY:0d694ff5e30274e5","status":"pending"}
11:29:16 [11:29:16.704]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f5feb6feb5fdb147 settled=false
{"provingJobId":"1:BASE_PARITY:f5feb6feb5fdb147","status":"pending"}
11:29:16 [11:29:16.705]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0d694ff5e30274e5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0d694ff5e30274e5"}
11:29:16 [11:29:16.705]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0d694ff5e30274e5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0d694ff5e30274e5"}
11:29:16 [11:29:16.718]
INFO:
prover-client:proving-broker Proving Broker started
11:29:16 [11:29:16.719]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b05ec5171a4cbedd epochNumber=1
{"provingJobId":"1:BASE_PARITY:b05ec5171a4cbedd"}
11:29:16 [11:29:16.720]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQBu7Ra/1 with map size 1073741824
11:29:16 [11:29:16.723]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:16 [11:29:16.748]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3cd86fdeb2805dbe epochNumber=2
{"provingJobId":"1:BASE_PARITY:3cd86fdeb2805dbe"}
11:29:16 [11:29:16.749]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQBu7Ra/2 with map size 1073741824
11:29:16 [11:29:16.751]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:16 [11:29:16.768]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b05ec5171a4cbedd type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b05ec5171a4cbedd"}
11:29:16 [11:29:16.774]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3cd86fdeb2805dbe type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3cd86fdeb2805dbe"}
11:29:16 [11:29:16.782]
INFO:
prover-client:proving-broker Proving Broker started
11:29:16 [11:29:16.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb646f51b21618dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:bb646f51b21618dd"}
11:29:16 [11:29:16.784]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvJtaFA/1 with map size 1073741824
11:29:16 [11:29:16.786]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:16 [11:29:16.804]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e10e0c5d64e9067d epochNumber=2
{"provingJobId":"1:BASE_PARITY:e10e0c5d64e9067d"}
11:29:16 [11:29:16.806]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvJtaFA/2 with map size 1073741824
11:29:16 [11:29:16.809]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:16 [11:29:16.829]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bb646f51b21618dd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bb646f51b21618dd"}
11:29:16 [11:29:16.829]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bb646f51b21618dd type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:bb646f51b21618dd"}
11:29:16 [11:29:16.840]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e10e0c5d64e9067d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e10e0c5d64e9067d"}
11:29:16 [11:29:16.854]
INFO:
prover-client:proving-broker Proving Broker started
11:29:16 [11:29:16.855]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4a62b28425ed7edb err=test error
{"provingJoId":"1:BASE_PARITY:4a62b28425ed7edb"}
11:29:16 [11:29:16.862]
INFO:
prover-client:proving-broker Proving Broker started
11:29:16 [11:29:16.863]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e0d80584ef58ecdd not found
{"provingJobId":"1:BASE_PARITY:e0d80584ef58ecdd"}
11:29:16 [11:29:16.865]
INFO:
prover-client:proving-broker Proving Broker started
11:29:16 [11:29:16.866]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56b556df38292961 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56b556df38292961"}
11:29:16 [11:29:16.866]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-teste5uw4W/1 with map size 1073741824
11:29:16 [11:29:16.870]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:16 [11:29:16.892]
INFO:
prover-client:proving-broker Proving Broker started
11:29:16 [11:29:16.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:afcfd21e72664ae6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:afcfd21e72664ae6"}
11:29:16 [11:29:16.893]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testapVYAu/1 with map size 1073741824
11:29:16 [11:29:16.895]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.021]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:afcfd21e72664ae6 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:afcfd21e72664ae6"}
11:29:17 [11:29:17.025]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a28df7d86f2627e epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a28df7d86f2627e"}
11:29:17 [11:29:17.026]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test869Z4B/1 with map size 1073741824
11:29:17 [11:29:17.029]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.152]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2a28df7d86f2627e timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2a28df7d86f2627e"}
11:29:17 [11:29:17.157]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:527f0d6ffc67d7b8 epochNumber=10
{"provingJobId":"1:BASE_PARITY:527f0d6ffc67d7b8"}
11:29:17 [11:29:17.158]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test869Z4B/10 with map size 1073741824
11:29:17 [11:29:17.160]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.203]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:17 [11:29:17.203]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
11:29:17 [11:29:17.308]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.309]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:696b379ccd22462e epochNumber=1
{"provingJobId":"1:BASE_PARITY:696b379ccd22462e"}
11:29:17 [11:29:17.310]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOQT17C/1 with map size 1073741824
11:29:17 [11:29:17.312]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.487]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:696b379ccd22462e timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:696b379ccd22462e"}
11:29:17 [11:29:17.537]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.538]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3de7ce5b3fc9fdc6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3de7ce5b3fc9fdc6"}
11:29:17 [11:29:17.540]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGtzrkQ/1 with map size 1073741824
11:29:17 [11:29:17.543]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.566]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3de7ce5b3fc9fdc6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:3de7ce5b3fc9fdc6"}
11:29:17 [11:29:17.570]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.570]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47f8fc5bd1b4c7d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:47f8fc5bd1b4c7d2"}
11:29:17 [11:29:17.571]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlDaaiN/1 with map size 1073741824
11:29:17 [11:29:17.575]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.594]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:47f8fc5bd1b4c7d2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:47f8fc5bd1b4c7d2"}
11:29:17 [11:29:17.595]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:47f8fc5bd1b4c7d2 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:47f8fc5bd1b4c7d2"}
11:29:17 [11:29:17.606]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.607]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:93edf6da6b675bc2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:93edf6da6b675bc2"}
11:29:17 [11:29:17.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvcouLs/1 with map size 1073741824
11:29:17 [11:29:17.613]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.629]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:93edf6da6b675bc2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:93edf6da6b675bc2"}
11:29:17 [11:29:17.637]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.638]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:774828af2ebf7431 epochNumber=1
{"provingJobId":"1:BASE_PARITY:774828af2ebf7431"}
11:29:17 [11:29:17.640]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0s0kA1/1 with map size 1073741824
11:29:17 [11:29:17.643]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.665]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:774828af2ebf7431 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:774828af2ebf7431"}
11:29:17 [11:29:17.666]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5bb8366ff637037 epochNumber=3
{"provingJobId":"1:BASE_PARITY:f5bb8366ff637037"}
11:29:17 [11:29:17.667]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test0s0kA1/3 with map size 1073741824
11:29:17 [11:29:17.669]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.689]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:17 [11:29:17.689]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
11:29:17 [11:29:17.712]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:774828af2ebf7431 err=test error
{"provingJoId":"1:BASE_PARITY:774828af2ebf7431"}
11:29:17 [11:29:17.716]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6sqerN/1 with map size 1073741824
11:29:17 [11:29:17.720]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.740]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6sqerN/2 with map size 1073741824
11:29:17 [11:29:17.745]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.762]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.764]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ec7770ac4e5a8271 settled=false
{"provingJobId":"1:BASE_PARITY:ec7770ac4e5a8271","status":"pending"}
11:29:17 [11:29:17.765]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:19fa3bd30a66d04f settled=false
{"provingJobId":"1:BASE_PARITY:19fa3bd30a66d04f","status":"pending"}
11:29:17 [11:29:17.771]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfX3eXy/1 with map size 1073741824
11:29:17 [11:29:17.775]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.792]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testfX3eXy/2 with map size 1073741824
11:29:17 [11:29:17.794]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.824]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.826]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:79cd40358d9a9826 settled=true
{"provingJobId":"1:BASE_PARITY:79cd40358d9a9826","status":"fulfilled"}
11:29:17 [11:29:17.828]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:6e48254e6c22761d settled=true
{"provingJobId":"2:BASE_PARITY:6e48254e6c22761d","status":"fulfilled"}
11:29:17 [11:29:17.835]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHpfH8N/1 with map size 1073741824
11:29:17 [11:29:17.837]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.862]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testHpfH8N/2 with map size 1073741824
11:29:17 [11:29:17.866]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.883]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.886]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f54b29bc4bdb95cb settled=true
{"provingJobId":"1:BASE_PARITY:f54b29bc4bdb95cb","status":"fulfilled"}
11:29:17 [11:29:17.889]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e30609e32402b074 settled=false
{"provingJobId":"1:BASE_PARITY:e30609e32402b074","status":"pending"}
11:29:17 [11:29:17.898]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.899]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f8b008d1eae98ff epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f8b008d1eae98ff"}
11:29:17 [11:29:17.900]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testu7H2tg/1 with map size 1073741824
11:29:17 [11:29:17.902]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.931]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.932]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00e579a5ee012c69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:00e579a5ee012c69"}
11:29:17 [11:29:17.933]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:00e579a5ee012c69: Error: db error: Error: db error
11:29:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:00e579a5ee012c69"}
11:29:17 [11:29:17.937]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.938]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a894eda471565b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a894eda471565b3"}
11:29:17 [11:29:17.939]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2vbCDb/1 with map size 1073741824
11:29:17 [11:29:17.942]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.961]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7a894eda471565b3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7a894eda471565b3"}
11:29:17 [11:29:17.962]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7a894eda471565b3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7a894eda471565b3"}
11:29:17 [11:29:17.973]
INFO:
prover-client:proving-broker Proving Broker started
11:29:17 [11:29:17.974]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:285888247bd01f4d epochNumber=1
{"provingJobId":"1:BASE_PARITY:285888247bd01f4d"}
11:29:17 [11:29:17.975]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXRhUBV/1 with map size 1073741824
11:29:17 [11:29:17.977]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:17 [11:29:17.993]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:285888247bd01f4d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:285888247bd01f4d"}
11:29:17 [11:29:17.994]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:285888247bd01f4d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:285888247bd01f4d"}
11:29:17 [11:29:17.995]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:285888247bd01f4d: Error: db error
11:29:17 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:285888247bd01f4d"}
11:29:18 [11:29:18.001]
INFO:
prover-client:proving-broker Proving Broker started
11:29:18 [11:29:18.002]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b511e86d66438098 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b511e86d66438098"}
11:29:18 [11:29:18.003]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyvsCYW/1 with map size 1073741824
11:29:18 [11:29:18.006]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:18 [11:29:18.023]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b511e86d66438098 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b511e86d66438098"}
11:29:18 [11:29:18.024]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b511e86d66438098 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b511e86d66438098"}
11:29:18 [11:29:18.033]
INFO:
prover-client:proving-broker Proving Broker started
11:29:18 [11:29:18.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2a1b6f1e1aeb82e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2a1b6f1e1aeb82e"}
11:29:18 [11:29:18.034]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvs5STH/1 with map size 1073741824
11:29:18 [11:29:18.036]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:18 [11:29:18.053]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b2a1b6f1e1aeb82e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b2a1b6f1e1aeb82e"}
11:29:18 [11:29:18.054]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b2a1b6f1e1aeb82e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b2a1b6f1e1aeb82e"}
11:29:18 [11:29:18.055]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:b2a1b6f1e1aeb82e jobErr=test error: Error: db error
11:29:18 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:b2a1b6f1e1aeb82e"}
11:29:18 [11:29:18.060]
INFO:
prover-client:proving-broker Proving Broker started
11:29:18 [11:29:18.061]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:27ad0b0380996b66 not found
{"provingJobId":"1:BASE_PARITY:27ad0b0380996b66"}
11:29:18 [11:29:18.068]
INFO:
prover-client:proving-broker Proving Broker started
11:29:18 [11:29:18.069]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1eeaf5a9a8ac457f err=test error
{"provingJoId":"1:BASE_PARITY:1eeaf5a9a8ac457f"}
11:29:18 [11:29:18.072]
INFO:
prover-client:proving-broker Proving Broker started
11:29:18 [11:29:18.083]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b627e48ea5a49ab1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b627e48ea5a49ab1"}
11:29:18 [11:29:18.085]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmGgYF7/1 with map size 1073741824
11:29:18 [11:29:18.087]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:18 [11:29:18.103]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b627e48ea5a49ab1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b627e48ea5a49ab1"}
11:29:18 [11:29:18.104]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b627e48ea5a49ab1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b627e48ea5a49ab1"}
11:29:18 [11:29:18.120]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:bc9bb491e3c385f1 epochNumber=2
{"provingJobId":"2:BASE_PARITY:bc9bb491e3c385f1"}
11:29:18 [11:29:18.121]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testmGgYF7/2 with map size 1073741824
11:29:18 [11:29:18.124]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:18 [11:29:18.140]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:bc9bb491e3c385f1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:bc9bb491e3c385f1"}
11:29:18 [11:29:18.141]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:bc9bb491e3c385f1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:bc9bb491e3c385f1"}
11:29:18 [11:29:18.209]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:20a65bdb58969dbc epochNumber=3
{"provingJobId":"3:BASE_PARITY:20a65bdb58969dbc"}
11:29:18 [11:29:18.210]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testmGgYF7/3 with map size 1073741824
11:29:18 [11:29:18.213]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:18 [11:29:18.251]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:18 [11:29:18.251]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
11:29:18 [11:29:18.297]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:336951585f9e5f4f epochNumber=4
{"provingJobId":"4:BASE_PARITY:336951585f9e5f4f"}
11:29:18 [11:29:18.299]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testmGgYF7/4 with map size 1073741824
11:29:18 [11:29:18.301]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:18 [11:29:18.331]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:18 [11:29:18.332]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
11:29:18 [11:29:18.379]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:2914f425d249b246 epochNumber=5
{"provingJobId":"5:BASE_PARITY:2914f425d249b246"}
11:29:18 [11:29:18.381]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testmGgYF7/5 with map size 1073741824
11:29:18 [11:29:18.384]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
11:29:18 [11:29:18.411]
INFO:
prover-client:proving-broker Cleaned up jobs=1
11:29:18 [11:29:18.411]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
11:29:18
PASS src/proving_broker/proving_broker.test.ts (
13.464 s)
11:29:18 ProvingBroker
11:29:18 Producer API
11:29:18
✓ refuses stale jobs (53 ms)
11:29:18
✓ enqueues jobs (4 ms)
11:29:18
✓ ignores duplicate jobs (3 ms)
11:29:18
✓ reports correct status when enqueuing repeat jobs (4 ms)
11:29:18
✓ throws an error in case of duplicate job IDs (28 ms)
11:29:18
✓ returns not-found status for non-existing jobs (2 ms)
11:29:18
✓ cancels jobs in queue (3 ms)
11:29:18
✓ cancels jobs in-progress (3 ms)
11:29:18
✓ returns job result if successful (3 ms)
11:29:18
✓ returns job error if failed (3 ms)
11:29:18
✓ correctly returns job status for concurrent writes (3 ms)
11:29:18
✓ refuses stale jobs (41 ms)
11:29:18
✓ enqueues jobs (37 ms)
11:29:18
✓ ignores duplicate jobs (28 ms)
11:29:18
✓ reports correct status when enqueuing repeat jobs (30 ms)
11:29:18
✓ throws an error in case of duplicate job IDs (26 ms)
11:29:18
✓ returns not-found status for non-existing jobs (3 ms)
11:29:18
✓ cancels jobs in queue (34 ms)
11:29:18
✓ cancels jobs in-progress (30 ms)
11:29:18
✓ returns job result if successful (21 ms)
11:29:18
✓ returns job error if failed (33 ms)
11:29:18
✓ correctly returns job status for concurrent writes (26 ms)
11:29:18 Consumer API
11:29:18
✓ returns undefined if no jobs are available (1 ms)
11:29:18
✓ returns jobs in priority order (1 ms)
11:29:18
✓ returns undefined if no jobs are available for the given allowList (1 ms)
11:29:18
✓ returns a job if it is in the allowList (2 ms)
11:29:18
✓ returns the most important job if it is in the allowList (2 ms)
11:29:18
✓ returns any job if filter is empty (2 ms)
11:29:18
✓ returns a new job when reporting job success (2 ms)
11:29:18
✓ returns a new job when reporting permanent error (2 ms)
11:29:18
✓ returns a new job when reporting retry-able error (2 ms)
11:29:18
✓ returns a new job when reporting progress if current one is cancelled (8 ms)
11:29:18
✓ returns a new job if job is already in progress elsewhere (1056 ms)
11:29:18
✓ avoids sending the same job to a new agent after a restart (1007 ms)
11:29:18
✓ avoids sending a completed job to a new agent after a restart (1006 ms)
11:29:18
✓ tracks job result if in progress (3 ms)
11:29:18
✓ tracks job result even if job is in queue (3 ms)
11:29:18
✓ ignores reported job error if unknown job (2 ms)
11:29:18
✓ ignores job result if unknown job (1 ms)
11:29:18
✓ returns undefined if no jobs are available (3 ms)
11:29:18
✓ returns jobs in priority order (29 ms)
11:29:18
✓ returns undefined if no jobs are available for the given allowList (19 ms)
11:29:18
✓ returns a job if it is in the allowList (61 ms)
11:29:18
✓ returns the most important job if it is in the allowList (65 ms)
11:29:18
✓ returns any job if filter is empty (62 ms)
11:29:18
✓ returns a new job when reporting job success (36 ms)
11:29:18
✓ returns a new job when reporting permanent error (35 ms)
11:29:18
✓ returns a new job when reporting retry-able error (32 ms)
11:29:18
✓ returns a new job when reporting progress if current one is cancelled (47 ms)
11:29:18
✓ returns a new job if job is already in progress elsewhere (1108 ms)
11:29:18
✓ avoids sending the same job to a new agent after a restart (1067 ms)
11:29:18
✓ avoids sending a completed job to a new agent after a restart (1060 ms)
11:29:18
✓ tracks job result if in progress (66 ms)
11:29:18
✓ tracks job result even if job is in queue (69 ms)
11:29:18
✓ ignores reported job error if unknown job (5 ms)
11:29:18
✓ ignores job result if unknown job (6 ms)
11:29:18 Timeouts
11:29:18
✓ tracks in progress jobs (1 ms)
11:29:18
✓ re-enqueues jobs that time out (108 ms)
11:29:18
✓ cancel stale jobs that time out (236 ms)
11:29:18
✓ keeps the jobs in progress while it is alive (204 ms)
11:29:18
✓ tracks in progress jobs (26 ms)
11:29:18
✓ re-enqueues jobs that time out (132 ms)
11:29:18
✓ cancel stale jobs that time out (283 ms)
11:29:18
✓ keeps the jobs in progress while it is alive (228 ms)
11:29:18 Retries
11:29:18
✓ retries jobs (9 ms)
11:29:18
✓ retries up to a maximum number of times (8 ms)
11:29:18
✓ passing retry=false does not retry (3 ms)
11:29:18
✓ does not retry if job is stale (28 ms)
11:29:18
✓ retries jobs (33 ms)
11:29:18
✓ retries up to a maximum number of times (34 ms)
11:29:18
✓ passing retry=false does not retry (32 ms)
11:29:18
✓ does not retry if job is stale (78 ms)
11:29:18 Database management
11:29:18
✓ re-enqueues proof requests on start (2 ms)
11:29:18
✓ restores proof results on start (1 ms)
11:29:18
✓ only re-enqueues unfinished jobs (2 ms)
11:29:18
✓ saves job when enqueued (4 ms)
11:29:18
✓ does not retain job if database fails to save (3 ms)
11:29:18
✓ saves job result (3 ms)
11:29:18
✓ saves result even if database fails to save (2 ms)
11:29:18
✓ saves job error (2 ms)
11:29:18
✓ saves job error even if database fails to save (3 ms)
11:29:18
✓ does not save job result if job is unknown (2 ms)
11:29:18
✓ does not save job error if job is unknown (1 ms)
11:29:18
✓ cleans up old jobs periodically (261 ms)
11:29:18
✓ re-enqueues proof requests on start (54 ms)
11:29:18
✓ restores proof results on start (63 ms)
11:29:18
✓ only re-enqueues unfinished jobs (62 ms)
11:29:18
✓ saves job when enqueued (33 ms)
11:29:18
✓ does not retain job if database fails to save (7 ms)
11:29:18
✓ saves job result (36 ms)
11:29:18
✓ saves result even if database fails to save (26 ms)
11:29:18
✓ saves job error (32 ms)
11:29:18
✓ saves job error even if database fails to save (26 ms)
11:29:18
✓ does not save job result if job is unknown (4 ms)
11:29:18
✓ does not save job error if job is unknown (7 ms)
11:29:18
✓ cleans up old jobs periodically (383 ms)
11:29:18
11:29:18
Test Suites: 1 passed, 1 total
11:29:18
Tests: 96 passed, 96 total
11:29:18
Snapshots: 0 total
11:29:18
Time: 13.587 s
11:29:18
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
11:29:18
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?