Parent Log:
http://ci.aztec-labs.com/2aac72a9f0a3e7d4
Command: b818050160cbcbd8 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/184d5f7d956dd7251670a2dd6728f313cfdf3caf
Env: REF_NAME=gh-readonly-queue/next/pr-14822-e56baa7f24bac54baf9e2f22f6f33ae6fa8b8c0f CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 13 21:17:37 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14822_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
21:17:41 [21:17:41.798]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.805]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0fb6445b935c61b5 epochNumber=42
{"provingJobId":"1:BASE_PARITY:0fb6445b935c61b5"}
21:17:41 [21:17:41.808]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:7a921efbb6f73855 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a921efbb6f73855"}
21:17:41 [21:17:41.840]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.841]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0364d8e6efc5e5e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0364d8e6efc5e5e2"}
21:17:41 [21:17:41.842]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b0c2b7e36f5db36 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b0c2b7e36f5db36"}
21:17:41 [21:17:41.844]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.845]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4d16f2653ef2955 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e4d16f2653ef2955"}
21:17:41 [21:17:41.846]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e4d16f2653ef2955 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e4d16f2653ef2955"}
21:17:41 [21:17:41.848]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.849]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a2908acd4a9a315 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2a2908acd4a9a315"}
21:17:41 [21:17:41.850]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2a2908acd4a9a315 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2a2908acd4a9a315"}
21:17:41 [21:17:41.851]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2a2908acd4a9a315 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2a2908acd4a9a315"}
21:17:41 [21:17:41.852]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2a2908acd4a9a315 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2a2908acd4a9a315"}
21:17:41 [21:17:41.853]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.853]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ecaee393a1e90916 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ecaee393a1e90916"}
21:17:41 [21:17:41.872]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.873]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b18b02cfea5a1280 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b18b02cfea5a1280"}
21:17:41 [21:17:41.874]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b18b02cfea5a1280
{"provingJobId":"1:BASE_PARITY:b18b02cfea5a1280"}
21:17:41 [21:17:41.874]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b18b02cfea5a1280 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b18b02cfea5a1280"}
21:17:41 [21:17:41.874]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b18b02cfea5a1280 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b18b02cfea5a1280"}
21:17:41 [21:17:41.876]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.876]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:481581b13ea8c4bb epochNumber=1
{"provingJobId":"1:BASE_PARITY:481581b13ea8c4bb"}
21:17:41 [21:17:41.877]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:481581b13ea8c4bb
{"provingJobId":"1:BASE_PARITY:481581b13ea8c4bb"}
21:17:41 [21:17:41.877]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:481581b13ea8c4bb type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:481581b13ea8c4bb"}
21:17:41 [21:17:41.878]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.879]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ef1681884b53c96 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ef1681884b53c96"}
21:17:41 [21:17:41.879]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6ef1681884b53c96 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6ef1681884b53c96"}
21:17:41 [21:17:41.880]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6ef1681884b53c96 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6ef1681884b53c96"}
21:17:41 [21:17:41.881]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.881]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fca6ea81b0cfd8ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:fca6ea81b0cfd8ca"}
21:17:41 [21:17:41.881]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fca6ea81b0cfd8ca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fca6ea81b0cfd8ca"}
21:17:41 [21:17:41.882]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fca6ea81b0cfd8ca type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fca6ea81b0cfd8ca"}
21:17:41 [21:17:41.882]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:786326a14afc1253 epochNumber=0
{"provingJobId":"1:BASE_PARITY:786326a14afc1253"}
21:17:41 [21:17:41.883]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:786326a14afc1253 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:786326a14afc1253"}
21:17:41 [21:17:41.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d9b75e15860ad7a epochNumber=0
{"provingJobId":"1:BASE_PARITY:2d9b75e15860ad7a"}
21:17:41 [21:17:41.883]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:786326a14afc1253 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:786326a14afc1253"}
21:17:41 [21:17:41.883]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a83c8f4a3603bbda epochNumber=0
{"provingJobId":"1:BASE_PARITY:a83c8f4a3603bbda"}
21:17:41 [21:17:41.884]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.891]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef07d7fdb1d25107 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ef07d7fdb1d25107"}
21:17:41 [21:17:41.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c549c2e7c577357 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c549c2e7c577357"}
21:17:41 [21:17:41.893]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.893]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f6a720adae8a9467 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f6a720adae8a9467"}
21:17:41 [21:17:41.894]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.895]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6079540ffebb4076 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6079540ffebb4076"}
21:17:41 [21:17:41.895]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9142b70f8192b6ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:9142b70f8192b6ea"}
21:17:41 [21:17:41.895]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:130a981d96f86769 epochNumber=2
{"provingJobId":"1:BASE_PARITY:130a981d96f86769"}
21:17:41 [21:17:41.895]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0c25bfa9e6150c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0c25bfa9e6150c9"}
21:17:41 [21:17:41.896]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.896]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b207cc2c2eaaf06f epochNumber=1
{"provingJobId":"1:BASE_PARITY:b207cc2c2eaaf06f"}
21:17:41 [21:17:41.897]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:124b7f030a62f46f epochNumber=1
{"provingJobId":"1:BASE_PARITY:124b7f030a62f46f"}
21:17:41 [21:17:41.897]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d20d538142e80d2d epochNumber=2
{"provingJobId":"1:BASE_PARITY:d20d538142e80d2d"}
21:17:41 [21:17:41.897]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f067883fb9aa0bd epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f067883fb9aa0bd"}
21:17:41 [21:17:41.898]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.899]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:29903ea63bea7419 epochNumber=1
{"provingJobId":"1:BASE_PARITY:29903ea63bea7419"}
21:17:41 [21:17:41.899]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cca08e707baa6d26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cca08e707baa6d26"}
21:17:41 [21:17:41.899]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:819a0fe1aad81e11 epochNumber=2
{"provingJobId":"1:BASE_PARITY:819a0fe1aad81e11"}
21:17:41 [21:17:41.899]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:709c3c95858eea12 epochNumber=1
{"provingJobId":"1:BASE_PARITY:709c3c95858eea12"}
21:17:41 [21:17:41.900]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.900]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6b6f07a02a4030d epochNumber=1
{"provingJobId":"1:BASE_PARITY:d6b6f07a02a4030d"}
21:17:41 [21:17:41.901]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a605dc42467d6620 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a605dc42467d6620"}
21:17:41 [21:17:41.901]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d6b6f07a02a4030d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d6b6f07a02a4030d"}
21:17:41 [21:17:41.902]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.902]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15fc5f915a509db8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:15fc5f915a509db8"}
21:17:41 [21:17:41.902]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:577c9ff0cf6ffbc4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:577c9ff0cf6ffbc4"}
21:17:41 [21:17:41.903]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:15fc5f915a509db8 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:15fc5f915a509db8"}
21:17:41 [21:17:41.903]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.904]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf5b6e2b278479fc epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf5b6e2b278479fc"}
21:17:41 [21:17:41.904]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5e7bca6d4dbf8fba epochNumber=1
{"provingJobId":"1:BASE_PARITY:5e7bca6d4dbf8fba"}
21:17:41 [21:17:41.904]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bf5b6e2b278479fc type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:bf5b6e2b278479fc"}
21:17:41 [21:17:41.905]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.905]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97e75d690484f711 epochNumber=1
{"provingJobId":"1:BASE_PARITY:97e75d690484f711"}
21:17:41 [21:17:41.906]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:97e75d690484f711
{"provingJobId":"1:BASE_PARITY:97e75d690484f711"}
21:17:41 [21:17:41.906]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:97e75d690484f711 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:97e75d690484f711"}
21:17:41 [21:17:41.906]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:922d1116db208d4f epochNumber=1
{"provingJobId":"1:BASE_PARITY:922d1116db208d4f"}
21:17:41 [21:17:41.906]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:97e75d690484f711 has already been completed
{"provingJobId":"1:BASE_PARITY:97e75d690484f711"}
21:17:41 [21:17:41.907]
INFO:
prover-client:proving-broker Proving Broker started
21:17:41 [21:17:41.908]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e5fbdcaa06bdee6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e5fbdcaa06bdee6"}
21:17:41 [21:17:41.908]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1eb6636f80be2aec epochNumber=2
{"provingJobId":"1:BASE_PARITY:1eb6636f80be2aec"}
21:17:42 [21:17:42.960]
INFO:
prover-client:proving-broker Proving Broker started
21:17:42 [21:17:42.960]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8e5fbdcaa06bdee6 settled=false
{"provingJobId":"1:BASE_PARITY:8e5fbdcaa06bdee6","status":"pending"}
21:17:42 [21:17:42.960]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1eb6636f80be2aec settled=false
{"provingJobId":"1:BASE_PARITY:1eb6636f80be2aec","status":"pending"}
21:17:42 [21:17:42.961]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8e5fbdcaa06bdee6 type=BASE_PARITY startedAt=1749849461908 older agent has taken job
{"provingJobId":"1:BASE_PARITY:8e5fbdcaa06bdee6"}
21:17:42 [21:17:42.961]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8e5fbdcaa06bdee6 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:8e5fbdcaa06bdee6"}
21:17:42 [21:17:42.962]
INFO:
prover-client:proving-broker Proving Broker started
21:17:42 [21:17:42.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6b6720c7d17a8ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6b6720c7d17a8ed"}
21:17:42 [21:17:42.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d42295814f7b990c epochNumber=2
{"provingJobId":"1:BASE_PARITY:d42295814f7b990c"}
21:17:43 [21:17:43.963]
INFO:
prover-client:proving-broker Proving Broker started
21:17:43 [21:17:43.963]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c6b6720c7d17a8ed settled=false
{"provingJobId":"1:BASE_PARITY:c6b6720c7d17a8ed","status":"pending"}
21:17:43 [21:17:43.964]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d42295814f7b990c settled=false
{"provingJobId":"1:BASE_PARITY:d42295814f7b990c","status":"pending"}
21:17:43 [21:17:43.964]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c6b6720c7d17a8ed type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:c6b6720c7d17a8ed"}
21:17:43 [21:17:43.966]
INFO:
prover-client:proving-broker Proving Broker started
21:17:43 [21:17:43.967]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a89d85ed135ca42e epochNumber=1
{"provingJobId":"1:BASE_PARITY:a89d85ed135ca42e"}
21:17:43 [21:17:43.967]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0682778ce610d740 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0682778ce610d740"}
21:17:44 [21:17:44.968]
INFO:
prover-client:proving-broker Proving Broker started
21:17:44 [21:17:44.969]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a89d85ed135ca42e settled=false
{"provingJobId":"1:BASE_PARITY:a89d85ed135ca42e","status":"pending"}
21:17:44 [21:17:44.969]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0682778ce610d740 settled=false
{"provingJobId":"1:BASE_PARITY:0682778ce610d740","status":"pending"}
21:17:44 [21:17:44.970]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a89d85ed135ca42e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a89d85ed135ca42e"}
21:17:44 [21:17:44.970]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a89d85ed135ca42e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a89d85ed135ca42e"}
21:17:44 [21:17:44.972]
INFO:
prover-client:proving-broker Proving Broker started
21:17:44 [21:17:44.973]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e01995f0f8575817 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e01995f0f8575817"}
21:17:44 [21:17:44.973]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce877876ec90cdc7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ce877876ec90cdc7"}
21:17:44 [21:17:44.974]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e01995f0f8575817 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e01995f0f8575817"}
21:17:44 [21:17:44.975]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ce877876ec90cdc7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ce877876ec90cdc7"}
21:17:44 [21:17:44.976]
INFO:
prover-client:proving-broker Proving Broker started
21:17:44 [21:17:44.977]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ab66c48d88c94a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ab66c48d88c94a4"}
21:17:44 [21:17:44.977]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a2fcd8db250a7d02 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a2fcd8db250a7d02"}
21:17:44 [21:17:44.977]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5ab66c48d88c94a4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5ab66c48d88c94a4"}
21:17:44 [21:17:44.977]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5ab66c48d88c94a4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5ab66c48d88c94a4"}
21:17:44 [21:17:44.978]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a2fcd8db250a7d02 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a2fcd8db250a7d02"}
21:17:44 [21:17:44.979]
INFO:
prover-client:proving-broker Proving Broker started
21:17:44 [21:17:44.979]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0209f0731b417902 err=test error
{"provingJoId":"1:BASE_PARITY:0209f0731b417902"}
21:17:44 [21:17:44.980]
INFO:
prover-client:proving-broker Proving Broker started
21:17:44 [21:17:44.980]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3e086ae5b023529c not found
{"provingJobId":"1:BASE_PARITY:3e086ae5b023529c"}
21:17:44 [21:17:44.981]
INFO:
prover-client:proving-broker Proving Broker started
21:17:44 [21:17:44.982]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50c6231ed4fc669f epochNumber=1
{"provingJobId":"1:BASE_PARITY:50c6231ed4fc669f"}
21:17:44 [21:17:44.983]
INFO:
prover-client:proving-broker Proving Broker started
21:17:44 [21:17:44.983]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b598cd01227be923 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b598cd01227be923"}
21:17:45 [21:17:45.084]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b598cd01227be923 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b598cd01227be923"}
21:17:45 [21:17:45.092]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.093]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7fc4153e66bc4a67 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7fc4153e66bc4a67"}
21:17:45 [21:17:45.194]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7fc4153e66bc4a67 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:7fc4153e66bc4a67"}
21:17:45 [21:17:45.201]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:586bb17d6d6591d9 epochNumber=10
{"provingJobId":"1:BASE_PARITY:586bb17d6d6591d9"}
21:17:45 [21:17:45.221]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:45 [21:17:45.329]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.330]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f7316d278a6b6d6c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f7316d278a6b6d6c"}
21:17:45 [21:17:45.483]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f7316d278a6b6d6c timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f7316d278a6b6d6c"}
21:17:45 [21:17:45.533]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae43f79f4c7b4130 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae43f79f4c7b4130"}
21:17:45 [21:17:45.535]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ae43f79f4c7b4130 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:ae43f79f4c7b4130"}
21:17:45 [21:17:45.536]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25c28300da7227e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:25c28300da7227e0"}
21:17:45 [21:17:45.537]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:25c28300da7227e0 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:25c28300da7227e0"}
21:17:45 [21:17:45.538]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:25c28300da7227e0 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:25c28300da7227e0"}
21:17:45 [21:17:45.539]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.540]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf86b8b45d85014c epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf86b8b45d85014c"}
21:17:45 [21:17:45.540]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bf86b8b45d85014c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bf86b8b45d85014c"}
21:17:45 [21:17:45.541]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.542]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0104f49d5d87ef71 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0104f49d5d87ef71"}
21:17:45 [21:17:45.542]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0104f49d5d87ef71 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0104f49d5d87ef71"}
21:17:45 [21:17:45.543]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68bbf98f1aa8d7c4 epochNumber=3
{"provingJobId":"1:BASE_PARITY:68bbf98f1aa8d7c4"}
21:17:45 [21:17:45.567]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:45 [21:17:45.569]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0104f49d5d87ef71 err=test error
{"provingJoId":"1:BASE_PARITY:0104f49d5d87ef71"}
21:17:45 [21:17:45.571]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.572]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:76735fdabf70a969 settled=false
{"provingJobId":"1:BASE_PARITY:76735fdabf70a969","status":"pending"}
21:17:45 [21:17:45.572]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8764f6638f10bd10 settled=false
{"provingJobId":"1:BASE_PARITY:8764f6638f10bd10","status":"pending"}
21:17:45 [21:17:45.574]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.575]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ad2f36b2c8f7c13a settled=true
{"provingJobId":"1:BASE_PARITY:ad2f36b2c8f7c13a","status":"fulfilled"}
21:17:45 [21:17:45.575]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:378fa46c52085ede settled=true
{"provingJobId":"2:BASE_PARITY:378fa46c52085ede","status":"fulfilled"}
21:17:45 [21:17:45.576]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.577]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b69a09b8ed29c80c settled=true
{"provingJobId":"1:BASE_PARITY:b69a09b8ed29c80c","status":"fulfilled"}
21:17:45 [21:17:45.577]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4b37ab3b6e3e1017 settled=false
{"provingJobId":"1:BASE_PARITY:4b37ab3b6e3e1017","status":"pending"}
21:17:45 [21:17:45.578]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.580]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a11999e4851d415 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a11999e4851d415"}
21:17:45 [21:17:45.582]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.582]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f60dc493458f11a epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f60dc493458f11a"}
21:17:45 [21:17:45.583]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:2f60dc493458f11a: Error: db error: Error: db error
21:17:45 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:2f60dc493458f11a"}
21:17:45 [21:17:45.584]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.585]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d820a518bfb924e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d820a518bfb924e6"}
21:17:45 [21:17:45.585]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d820a518bfb924e6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d820a518bfb924e6"}
21:17:45 [21:17:45.585]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d820a518bfb924e6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d820a518bfb924e6"}
21:17:45 [21:17:45.587]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.587]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c8fea62bc6344d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c8fea62bc6344d2"}
21:17:45 [21:17:45.588]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2c8fea62bc6344d2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2c8fea62bc6344d2"}
21:17:45 [21:17:45.588]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2c8fea62bc6344d2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2c8fea62bc6344d2"}
21:17:45 [21:17:45.589]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:2c8fea62bc6344d2: Error: db error
21:17:45 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:2c8fea62bc6344d2"}
21:17:45 [21:17:45.590]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.590]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68dd23c499640061 epochNumber=1
{"provingJobId":"1:BASE_PARITY:68dd23c499640061"}
21:17:45 [21:17:45.591]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:68dd23c499640061 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:68dd23c499640061"}
21:17:45 [21:17:45.591]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:68dd23c499640061 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:68dd23c499640061"}
21:17:45 [21:17:45.592]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.593]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c7c0505044a1703f epochNumber=1
{"provingJobId":"1:BASE_PARITY:c7c0505044a1703f"}
21:17:45 [21:17:45.593]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c7c0505044a1703f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c7c0505044a1703f"}
21:17:45 [21:17:45.593]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c7c0505044a1703f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:c7c0505044a1703f"}
21:17:45 [21:17:45.594]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:c7c0505044a1703f jobErr=test error: Error: db error
21:17:45 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:c7c0505044a1703f"}
21:17:45 [21:17:45.595]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.596]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:39744c0bc755d6b8 not found
{"provingJobId":"1:BASE_PARITY:39744c0bc755d6b8"}
21:17:45 [21:17:45.597]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.598]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3fd10bd2a4a3e7dc err=test error
{"provingJoId":"1:BASE_PARITY:3fd10bd2a4a3e7dc"}
21:17:45 [21:17:45.599]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.610]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b41d058053c7240 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b41d058053c7240"}
21:17:45 [21:17:45.611]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4b41d058053c7240 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4b41d058053c7240"}
21:17:45 [21:17:45.611]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4b41d058053c7240 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4b41d058053c7240"}
21:17:45 [21:17:45.622]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:1b0f169a79ad1c44 epochNumber=2
{"provingJobId":"2:BASE_PARITY:1b0f169a79ad1c44"}
21:17:45 [21:17:45.623]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:1b0f169a79ad1c44 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:1b0f169a79ad1c44"}
21:17:45 [21:17:45.623]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:1b0f169a79ad1c44 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:1b0f169a79ad1c44"}
21:17:45 [21:17:45.685]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:1f3bc66aecf801bd epochNumber=3
{"provingJobId":"3:BASE_PARITY:1f3bc66aecf801bd"}
21:17:45 [21:17:45.701]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:45 [21:17:45.746]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:ba8a809a90e293c2 epochNumber=4
{"provingJobId":"4:BASE_PARITY:ba8a809a90e293c2"}
21:17:45 [21:17:45.753]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:45 [21:17:45.808]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:c11523dddbc3b3b8 epochNumber=5
{"provingJobId":"5:BASE_PARITY:c11523dddbc3b3b8"}
21:17:45 [21:17:45.829]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:45 [21:17:45.862]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.863]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9724fb48a78dd793 epochNumber=42
{"provingJobId":"1:BASE_PARITY:9724fb48a78dd793"}
21:17:45 [21:17:45.864]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testLwAvoZ/42 with map size 1073741824
21:17:45 [21:17:45.867]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:45 [21:17:45.889]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:ef977397b0512af9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef977397b0512af9"}
21:17:45 [21:17:45.893]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.893]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b56e4d1c10037ee8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b56e4d1c10037ee8"}
21:17:45 [21:17:45.894]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testD3zbCh/1 with map size 1073741824
21:17:45 [21:17:45.896]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:45 [21:17:45.916]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9bbecbc6c2c29d16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9bbecbc6c2c29d16"}
21:17:45 [21:17:45.931]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.932]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:358ed542581ed20e epochNumber=1
{"provingJobId":"1:BASE_PARITY:358ed542581ed20e"}
21:17:45 [21:17:45.934]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSomSmO/1 with map size 1073741824
21:17:45 [21:17:45.936]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:45 [21:17:45.953]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:358ed542581ed20e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:358ed542581ed20e"}
21:17:45 [21:17:45.957]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.958]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3aa119a23dd290df epochNumber=1
{"provingJobId":"1:BASE_PARITY:3aa119a23dd290df"}
21:17:45 [21:17:45.959]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testL7wL8G/1 with map size 1073741824
21:17:45 [21:17:45.961]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:45 [21:17:45.979]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3aa119a23dd290df epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3aa119a23dd290df"}
21:17:45 [21:17:45.980]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3aa119a23dd290df type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3aa119a23dd290df"}
21:17:45 [21:17:45.986]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3aa119a23dd290df epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3aa119a23dd290df"}
21:17:45 [21:17:45.990]
INFO:
prover-client:proving-broker Proving Broker started
21:17:45 [21:17:45.990]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:73992cb48395388f epochNumber=1
{"provingJobId":"1:BASE_PARITY:73992cb48395388f"}
21:17:45 [21:17:45.991]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGP8Tvo/1 with map size 1073741824
21:17:45 [21:17:45.993]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.037]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.042]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.043]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:37523f949350fc69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:37523f949350fc69"}
21:17:46 [21:17:46.044]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFlIKtS/1 with map size 1073741824
21:17:46 [21:17:46.046]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.067]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:37523f949350fc69
{"provingJobId":"1:BASE_PARITY:37523f949350fc69"}
21:17:46 [21:17:46.068]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:37523f949350fc69 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:37523f949350fc69"}
21:17:46 [21:17:46.068]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:37523f949350fc69 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:37523f949350fc69"}
21:17:46 [21:17:46.079]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ba15a1792d145b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0ba15a1792d145b9"}
21:17:46 [21:17:46.080]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRINY1Y/1 with map size 1073741824
21:17:46 [21:17:46.083]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.102]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0ba15a1792d145b9
{"provingJobId":"1:BASE_PARITY:0ba15a1792d145b9"}
21:17:46 [21:17:46.102]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0ba15a1792d145b9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:0ba15a1792d145b9"}
21:17:46 [21:17:46.113]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.113]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f8534fbaf7e6fd3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f8534fbaf7e6fd3"}
21:17:46 [21:17:46.114]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZSQDfJ/1 with map size 1073741824
21:17:46 [21:17:46.122]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.142]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9f8534fbaf7e6fd3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9f8534fbaf7e6fd3"}
21:17:46 [21:17:46.143]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9f8534fbaf7e6fd3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9f8534fbaf7e6fd3"}
21:17:46 [21:17:46.151]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.152]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7743e9f45ac65184 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7743e9f45ac65184"}
21:17:46 [21:17:46.153]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcrvKUF/1 with map size 1073741824
21:17:46 [21:17:46.157]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.174]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7743e9f45ac65184 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7743e9f45ac65184"}
21:17:46 [21:17:46.174]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7743e9f45ac65184 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:7743e9f45ac65184"}
21:17:46 [21:17:46.183]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.184]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b244b84e30e2e3ed epochNumber=0
{"provingJobId":"1:BASE_PARITY:b244b84e30e2e3ed"}
21:17:46 [21:17:46.185]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test398fkL/0 with map size 1073741824
21:17:46 [21:17:46.187]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.205]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b244b84e30e2e3ed epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b244b84e30e2e3ed"}
21:17:46 [21:17:46.206]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:953591e07c6ccce1 epochNumber=0
{"provingJobId":"1:BASE_PARITY:953591e07c6ccce1"}
21:17:46 [21:17:46.206]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b244b84e30e2e3ed epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b244b84e30e2e3ed"}
21:17:46 [21:17:46.207]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60bc0d34dbb8458c epochNumber=0
{"provingJobId":"1:BASE_PARITY:60bc0d34dbb8458c"}
21:17:46 [21:17:46.225]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.230]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.230]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1309927b0febe517 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1309927b0febe517"}
21:17:46 [21:17:46.231]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpe5ghd/2 with map size 1073741824
21:17:46 [21:17:46.236]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.259]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de0f3e5946c66f83 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de0f3e5946c66f83"}
21:17:46 [21:17:46.260]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpe5ghd/1 with map size 1073741824
21:17:46 [21:17:46.262]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.283]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.285]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b334de2836b0e708 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b334de2836b0e708"}
21:17:46 [21:17:46.286]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testb2JUlH/1 with map size 1073741824
21:17:46 [21:17:46.288]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.309]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.310]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47ae951e07b66673 epochNumber=1
{"provingJobId":"1:BASE_PARITY:47ae951e07b66673"}
21:17:46 [21:17:46.310]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSHWGJP/1 with map size 1073741824
21:17:46 [21:17:46.312]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.337]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c425109597afacf5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c425109597afacf5"}
21:17:46 [21:17:46.343]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12d1a7b2e07e50dd epochNumber=2
{"provingJobId":"1:BASE_PARITY:12d1a7b2e07e50dd"}
21:17:46 [21:17:46.345]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSHWGJP/2 with map size 1073741824
21:17:46 [21:17:46.348]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.367]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab61371914f0ac48 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab61371914f0ac48"}
21:17:46 [21:17:46.377]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.378]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ac13704c4a45f97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3ac13704c4a45f97"}
21:17:46 [21:17:46.378]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1XhJyz/1 with map size 1073741824
21:17:46 [21:17:46.381]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.400]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d8cb22d01fa5ff60 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d8cb22d01fa5ff60"}
21:17:46 [21:17:46.407]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d72d6f756194a94f epochNumber=2
{"provingJobId":"1:BASE_PARITY:d72d6f756194a94f"}
21:17:46 [21:17:46.408]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test1XhJyz/2 with map size 1073741824
21:17:46 [21:17:46.410]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:795e4abf174eb915 epochNumber=1
{"provingJobId":"1:BASE_PARITY:795e4abf174eb915"}
21:17:46 [21:17:46.448]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.449]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9224bb715f99adff epochNumber=1
{"provingJobId":"1:BASE_PARITY:9224bb715f99adff"}
21:17:46 [21:17:46.450]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkoc1qY/1 with map size 1073741824
21:17:46 [21:17:46.452]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.472]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b109a83f7cb165b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b109a83f7cb165b6"}
21:17:46 [21:17:46.480]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b7cd29ccc48ec4b epochNumber=2
{"provingJobId":"1:BASE_PARITY:0b7cd29ccc48ec4b"}
21:17:46 [21:17:46.481]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkoc1qY/2 with map size 1073741824
21:17:46 [21:17:46.484]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.503]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:78f3766482a16f06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:78f3766482a16f06"}
21:17:46 [21:17:46.513]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.515]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f961c51f81cf82dd epochNumber=1
{"provingJobId":"1:BASE_PARITY:f961c51f81cf82dd"}
21:17:46 [21:17:46.516]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyq7gvJ/1 with map size 1073741824
21:17:46 [21:17:46.521]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.546]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6321ea4d93e4251e epochNumber=1
{"provingJobId":"1:BASE_PARITY:6321ea4d93e4251e"}
21:17:46 [21:17:46.553]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f961c51f81cf82dd type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f961c51f81cf82dd"}
21:17:46 [21:17:46.564]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.565]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:853bb8a8949bc64e epochNumber=1
{"provingJobId":"1:BASE_PARITY:853bb8a8949bc64e"}
21:17:46 [21:17:46.565]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoCP4C0/1 with map size 1073741824
21:17:46 [21:17:46.568]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.587]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21bd6d490d57f6c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:21bd6d490d57f6c0"}
21:17:46 [21:17:46.594]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:853bb8a8949bc64e type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:853bb8a8949bc64e"}
21:17:46 [21:17:46.605]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f379ff59fcfc49e epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f379ff59fcfc49e"}
21:17:46 [21:17:46.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVIaKnn/1 with map size 1073741824
21:17:46 [21:17:46.610]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.637]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:378b9d8a2ab53e24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:378b9d8a2ab53e24"}
21:17:46 [21:17:46.644]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9f379ff59fcfc49e type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:9f379ff59fcfc49e"}
21:17:46 [21:17:46.648]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.649]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79be1c2eb9b674d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:79be1c2eb9b674d7"}
21:17:46 [21:17:46.650]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIJJOsk/1 with map size 1073741824
21:17:46 [21:17:46.653]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.673]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:79be1c2eb9b674d7
{"provingJobId":"1:BASE_PARITY:79be1c2eb9b674d7"}
21:17:46 [21:17:46.674]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:79be1c2eb9b674d7 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:79be1c2eb9b674d7"}
21:17:46 [21:17:46.680]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57d9f9b44d417e1c epochNumber=1
{"provingJobId":"1:BASE_PARITY:57d9f9b44d417e1c"}
21:17:46 [21:17:46.690]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:79be1c2eb9b674d7 has already been completed
{"provingJobId":"1:BASE_PARITY:79be1c2eb9b674d7"}
21:17:46 [21:17:46.694]
INFO:
prover-client:proving-broker Proving Broker started
21:17:46 [21:17:46.695]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:abfdf96e78f5eed6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:abfdf96e78f5eed6"}
21:17:46 [21:17:46.696]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9Ska2j/1 with map size 1073741824
21:17:46 [21:17:46.698]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:46 [21:17:46.721]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e40999af55b9ce0e epochNumber=2
{"provingJobId":"1:BASE_PARITY:e40999af55b9ce0e"}
21:17:46 [21:17:46.722]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test9Ska2j/2 with map size 1073741824
21:17:46 [21:17:46.733]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:47 [21:17:47.802]
INFO:
prover-client:proving-broker Proving Broker started
21:17:47 [21:17:47.808]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:abfdf96e78f5eed6 settled=false
{"provingJobId":"1:BASE_PARITY:abfdf96e78f5eed6","status":"pending"}
21:17:47 [21:17:47.809]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e40999af55b9ce0e settled=false
{"provingJobId":"1:BASE_PARITY:e40999af55b9ce0e","status":"pending"}
21:17:47 [21:17:47.810]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:abfdf96e78f5eed6 type=BASE_PARITY startedAt=1749849466750 older agent has taken job
{"provingJobId":"1:BASE_PARITY:abfdf96e78f5eed6"}
21:17:47 [21:17:47.810]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:abfdf96e78f5eed6 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:abfdf96e78f5eed6"}
21:17:47 [21:17:47.817]
INFO:
prover-client:proving-broker Proving Broker started
21:17:47 [21:17:47.818]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:abb5dda3b097d9e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:abb5dda3b097d9e7"}
21:17:47 [21:17:47.819]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvDvcEB/1 with map size 1073741824
21:17:47 [21:17:47.822]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:47 [21:17:47.837]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a09a67995ec8b31 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7a09a67995ec8b31"}
21:17:47 [21:17:47.838]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testvDvcEB/2 with map size 1073741824
21:17:47 [21:17:47.840]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:48 [21:17:48.856]
INFO:
prover-client:proving-broker Proving Broker started
21:17:48 [21:17:48.858]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:abb5dda3b097d9e7 settled=false
{"provingJobId":"1:BASE_PARITY:abb5dda3b097d9e7","status":"pending"}
21:17:48 [21:17:48.860]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7a09a67995ec8b31 settled=false
{"provingJobId":"1:BASE_PARITY:7a09a67995ec8b31","status":"pending"}
21:17:48 [21:17:48.861]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:abb5dda3b097d9e7 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:abb5dda3b097d9e7"}
21:17:48 [21:17:48.864]
INFO:
prover-client:proving-broker Proving Broker started
21:17:48 [21:17:48.866]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1a1000760d51a87 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1a1000760d51a87"}
21:17:48 [21:17:48.867]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKq1aQk/1 with map size 1073741824
21:17:48 [21:17:48.870]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:48 [21:17:48.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d88de13d1774ade3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d88de13d1774ade3"}
21:17:48 [21:17:48.894]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKq1aQk/2 with map size 1073741824
21:17:48 [21:17:48.900]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:49 [21:17:49.927]
INFO:
prover-client:proving-broker Proving Broker started
21:17:49 [21:17:49.929]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e1a1000760d51a87 settled=false
{"provingJobId":"1:BASE_PARITY:e1a1000760d51a87","status":"pending"}
21:17:49 [21:17:49.931]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d88de13d1774ade3 settled=false
{"provingJobId":"1:BASE_PARITY:d88de13d1774ade3","status":"pending"}
21:17:49 [21:17:49.932]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e1a1000760d51a87 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e1a1000760d51a87"}
21:17:49 [21:17:49.932]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e1a1000760d51a87 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e1a1000760d51a87"}
21:17:49 [21:17:49.941]
INFO:
prover-client:proving-broker Proving Broker started
21:17:49 [21:17:49.941]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:117ae505af028a41 epochNumber=1
{"provingJobId":"1:BASE_PARITY:117ae505af028a41"}
21:17:49 [21:17:49.942]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testufPiM7/1 with map size 1073741824
21:17:49 [21:17:49.944]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:49 [21:17:49.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b4985ab3fa0eee7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:8b4985ab3fa0eee7"}
21:17:49 [21:17:49.963]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testufPiM7/2 with map size 1073741824
21:17:49 [21:17:49.966]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:49 [21:17:49.991]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:117ae505af028a41 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:117ae505af028a41"}
21:17:49 [21:17:49.998]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8b4985ab3fa0eee7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8b4985ab3fa0eee7"}
21:17:50 [21:17:50.011]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.011]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f4b8b63269aaad4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f4b8b63269aaad4"}
21:17:50 [21:17:50.012]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAMUS7A/1 with map size 1073741824
21:17:50 [21:17:50.014]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.039]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57f8328efe2d81ff epochNumber=2
{"provingJobId":"1:BASE_PARITY:57f8328efe2d81ff"}
21:17:50 [21:17:50.040]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAMUS7A/2 with map size 1073741824
21:17:50 [21:17:50.042]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.065]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5f4b8b63269aaad4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5f4b8b63269aaad4"}
21:17:50 [21:17:50.066]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5f4b8b63269aaad4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5f4b8b63269aaad4"}
21:17:50 [21:17:50.073]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:57f8328efe2d81ff type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:57f8328efe2d81ff"}
21:17:50 [21:17:50.084]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.085]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6ab5aa3f5e1c36b7 err=test error
{"provingJoId":"1:BASE_PARITY:6ab5aa3f5e1c36b7"}
21:17:50 [21:17:50.088]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.089]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d5f90d98d292dd1f not found
{"provingJobId":"1:BASE_PARITY:d5f90d98d292dd1f"}
21:17:50 [21:17:50.093]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.094]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0148cb2202c93a02 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0148cb2202c93a02"}
21:17:50 [21:17:50.095]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOvSdh6/1 with map size 1073741824
21:17:50 [21:17:50.097]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.119]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.120]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6af6f8873dd119af epochNumber=1
{"provingJobId":"1:BASE_PARITY:6af6f8873dd119af"}
21:17:50 [21:17:50.120]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxujhPo/1 with map size 1073741824
21:17:50 [21:17:50.123]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.247]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6af6f8873dd119af timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6af6f8873dd119af"}
21:17:50 [21:17:50.260]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.262]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a5c70f59620271f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a5c70f59620271f"}
21:17:50 [21:17:50.263]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIyYsN3/1 with map size 1073741824
21:17:50 [21:17:50.275]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.417]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4a5c70f59620271f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4a5c70f59620271f"}
21:17:50 [21:17:50.418]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fece49f9bbffd49b epochNumber=10
{"provingJobId":"1:BASE_PARITY:fece49f9bbffd49b"}
21:17:50 [21:17:50.419]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testIyYsN3/10 with map size 1073741824
21:17:50 [21:17:50.426]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.469]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:50 [21:17:50.471]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:17:50 [21:17:50.587]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.588]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85c177c03f515adf epochNumber=1
{"provingJobId":"1:BASE_PARITY:85c177c03f515adf"}
21:17:50 [21:17:50.589]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTu610M/1 with map size 1073741824
21:17:50 [21:17:50.593]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.771]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:85c177c03f515adf timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:85c177c03f515adf"}
21:17:50 [21:17:50.827]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.829]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9a5484d0f1493c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9a5484d0f1493c8"}
21:17:50 [21:17:50.831]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhr8Oa0/1 with map size 1073741824
21:17:50 [21:17:50.843]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.866]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b9a5484d0f1493c8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b9a5484d0f1493c8"}
21:17:50 [21:17:50.871]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:30ce72b89cf88386 epochNumber=1
{"provingJobId":"1:BASE_PARITY:30ce72b89cf88386"}
21:17:50 [21:17:50.874]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWFZmEY/1 with map size 1073741824
21:17:50 [21:17:50.879]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.900]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:30ce72b89cf88386 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:30ce72b89cf88386"}
21:17:50 [21:17:50.902]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:30ce72b89cf88386 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:30ce72b89cf88386"}
21:17:50 [21:17:50.915]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.917]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6bc0578c2b2bdf38 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6bc0578c2b2bdf38"}
21:17:50 [21:17:50.921]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5FBFX8/1 with map size 1073741824
21:17:50 [21:17:50.927]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.953]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6bc0578c2b2bdf38 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6bc0578c2b2bdf38"}
21:17:50 [21:17:50.965]
INFO:
prover-client:proving-broker Proving Broker started
21:17:50 [21:17:50.967]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:171f91aa8f733dc4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:171f91aa8f733dc4"}
21:17:50 [21:17:50.968]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGgZfA8/1 with map size 1073741824
21:17:50 [21:17:50.972]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:50 [21:17:50.995]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:171f91aa8f733dc4 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:171f91aa8f733dc4"}
21:17:50 [21:17:50.998]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8583ae9e19272467 epochNumber=3
{"provingJobId":"1:BASE_PARITY:8583ae9e19272467"}
21:17:50 [21:17:51.000]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testGgZfA8/3 with map size 1073741824
21:17:51 [21:17:51.004]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.043]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:51 [21:17:51.043]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:17:51 [21:17:51.055]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:171f91aa8f733dc4 err=test error
{"provingJoId":"1:BASE_PARITY:171f91aa8f733dc4"}
21:17:51 [21:17:51.065]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKhZOdB/1 with map size 1073741824
21:17:51 [21:17:51.070]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.099]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKhZOdB/2 with map size 1073741824
21:17:51 [21:17:51.105]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.129]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.132]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ce405d34e4c6ac06 settled=false
{"provingJobId":"1:BASE_PARITY:ce405d34e4c6ac06","status":"pending"}
21:17:51 [21:17:51.136]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f74870edeca3ac58 settled=false
{"provingJobId":"1:BASE_PARITY:f74870edeca3ac58","status":"pending"}
21:17:51 [21:17:51.143]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZGcBQZ/1 with map size 1073741824
21:17:51 [21:17:51.146]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.167]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZGcBQZ/2 with map size 1073741824
21:17:51 [21:17:51.172]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.209]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.212]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ee10aaad612f03a5 settled=true
{"provingJobId":"1:BASE_PARITY:ee10aaad612f03a5","status":"fulfilled"}
21:17:51 [21:17:51.214]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:1bfec0dd5bddbacc settled=true
{"provingJobId":"2:BASE_PARITY:1bfec0dd5bddbacc","status":"fulfilled"}
21:17:51 [21:17:51.223]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2KZauY/1 with map size 1073741824
21:17:51 [21:17:51.230]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.256]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2KZauY/2 with map size 1073741824
21:17:51 [21:17:51.263]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.285]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.287]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:781dd119955f9a6b settled=true
{"provingJobId":"1:BASE_PARITY:781dd119955f9a6b","status":"fulfilled"}
21:17:51 [21:17:51.291]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:40b67cb20b317b5c settled=false
{"provingJobId":"1:BASE_PARITY:40b67cb20b317b5c","status":"pending"}
21:17:51 [21:17:51.298]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.299]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe8729478320c420 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe8729478320c420"}
21:17:51 [21:17:51.301]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuwrZsP/1 with map size 1073741824
21:17:51 [21:17:51.306]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.329]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.330]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4834116187c19655 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4834116187c19655"}
21:17:51 [21:17:51.331]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:4834116187c19655: Error: db error: Error: db error
21:17:51 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:4834116187c19655"}
21:17:51 [21:17:51.335]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.335]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58723a8e284ade3d epochNumber=1
{"provingJobId":"1:BASE_PARITY:58723a8e284ade3d"}
21:17:51 [21:17:51.336]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrKKsvb/1 with map size 1073741824
21:17:51 [21:17:51.339]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.354]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:58723a8e284ade3d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:58723a8e284ade3d"}
21:17:51 [21:17:51.355]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:58723a8e284ade3d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:58723a8e284ade3d"}
21:17:51 [21:17:51.367]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.368]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:868db95b4d5d781c epochNumber=1
{"provingJobId":"1:BASE_PARITY:868db95b4d5d781c"}
21:17:51 [21:17:51.371]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSMaQkn/1 with map size 1073741824
21:17:51 [21:17:51.373]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.396]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:868db95b4d5d781c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:868db95b4d5d781c"}
21:17:51 [21:17:51.396]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:868db95b4d5d781c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:868db95b4d5d781c"}
21:17:51 [21:17:51.397]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:868db95b4d5d781c: Error: db error
21:17:51 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:868db95b4d5d781c"}
21:17:51 [21:17:51.403]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.404]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70ded9e699c6c0b6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:70ded9e699c6c0b6"}
21:17:51 [21:17:51.406]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4Sy28k/1 with map size 1073741824
21:17:51 [21:17:51.410]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.436]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:70ded9e699c6c0b6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:70ded9e699c6c0b6"}
21:17:51 [21:17:51.436]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:70ded9e699c6c0b6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:70ded9e699c6c0b6"}
21:17:51 [21:17:51.455]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.456]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2747f69c3d564581 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2747f69c3d564581"}
21:17:51 [21:17:51.457]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testz68MSI/1 with map size 1073741824
21:17:51 [21:17:51.460]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.484]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2747f69c3d564581 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2747f69c3d564581"}
21:17:51 [21:17:51.484]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2747f69c3d564581 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2747f69c3d564581"}
21:17:51 [21:17:51.485]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:2747f69c3d564581 jobErr=test error: Error: db error
21:17:51 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:2747f69c3d564581"}
21:17:51 [21:17:51.490]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.491]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7f41b2d469275fe1 not found
{"provingJobId":"1:BASE_PARITY:7f41b2d469275fe1"}
21:17:51 [21:17:51.494]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.495]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f4c536609ad65b0c err=test error
{"provingJoId":"1:BASE_PARITY:f4c536609ad65b0c"}
21:17:51 [21:17:51.500]
INFO:
prover-client:proving-broker Proving Broker started
21:17:51 [21:17:51.512]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ade5d19ab344f0f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ade5d19ab344f0f9"}
21:17:51 [21:17:51.513]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuOmML6/1 with map size 1073741824
21:17:51 [21:17:51.515]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.537]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ade5d19ab344f0f9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ade5d19ab344f0f9"}
21:17:51 [21:17:51.537]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ade5d19ab344f0f9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ade5d19ab344f0f9"}
21:17:51 [21:17:51.555]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:78cd2deb3ef1f6b8 epochNumber=2
{"provingJobId":"2:BASE_PARITY:78cd2deb3ef1f6b8"}
21:17:51 [21:17:51.556]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testuOmML6/2 with map size 1073741824
21:17:51 [21:17:51.559]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.581]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:78cd2deb3ef1f6b8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:78cd2deb3ef1f6b8"}
21:17:51 [21:17:51.581]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:78cd2deb3ef1f6b8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:78cd2deb3ef1f6b8"}
21:17:51 [21:17:51.653]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:da149715c1276d04 epochNumber=3
{"provingJobId":"3:BASE_PARITY:da149715c1276d04"}
21:17:51 [21:17:51.654]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testuOmML6/3 with map size 1073741824
21:17:51 [21:17:51.656]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.679]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:51 [21:17:51.680]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
21:17:51 [21:17:51.736]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:08c50f29fd9afd94 epochNumber=4
{"provingJobId":"4:BASE_PARITY:08c50f29fd9afd94"}
21:17:51 [21:17:51.738]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testuOmML6/4 with map size 1073741824
21:17:51 [21:17:51.740]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.758]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:51 [21:17:51.758]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
21:17:51 [21:17:51.817]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:867a535a58bd71b4 epochNumber=5
{"provingJobId":"5:BASE_PARITY:867a535a58bd71b4"}
21:17:51 [21:17:51.819]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testuOmML6/5 with map size 1073741824
21:17:51 [21:17:51.823]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
21:17:51 [21:17:51.862]
INFO:
prover-client:proving-broker Cleaned up jobs=1
21:17:51 [21:17:51.863]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
21:17:51
PASS src/proving_broker/proving_broker.test.ts (
13.195 s)
21:17:51 ProvingBroker
21:17:51 Producer API
21:17:51
✓ refuses stale jobs (45 ms)
21:17:51
✓ enqueues jobs (5 ms)
21:17:51
✓ ignores duplicate jobs (4 ms)
21:17:51
✓ reports correct status when enqueuing repeat jobs (4 ms)
21:17:51
✓ throws an error in case of duplicate job IDs (19 ms)
21:17:51
✓ returns not-found status for non-existing jobs (1 ms)
21:17:51
✓ cancels jobs in queue (2 ms)
21:17:51
✓ cancels jobs in-progress (2 ms)
21:17:51
✓ returns job result if successful (2 ms)
21:17:51
✓ returns job error if failed (2 ms)
21:17:51
✓ correctly returns job status for concurrent writes (2 ms)
21:17:51
✓ refuses stale jobs (31 ms)
21:17:51
✓ enqueues jobs (36 ms)
21:17:51
✓ ignores duplicate jobs (27 ms)
21:17:51
✓ reports correct status when enqueuing repeat jobs (33 ms)
21:17:51
✓ throws an error in case of duplicate job IDs (34 ms)
21:17:51
✓ returns not-found status for non-existing jobs (16 ms)
21:17:51
✓ cancels jobs in queue (36 ms)
21:17:51
✓ cancels jobs in-progress (34 ms)
21:17:51
✓ returns job result if successful (39 ms)
21:17:51
✓ returns job error if failed (32 ms)
21:17:51
✓ correctly returns job status for concurrent writes (40 ms)
21:17:51 Consumer API
21:17:51
✓ returns undefined if no jobs are available (7 ms)
21:17:51
✓ returns jobs in priority order (2 ms)
21:17:51
✓ returns undefined if no jobs are available for the given allowList (1 ms)
21:17:51
✓ returns a job if it is in the allowList (2 ms)
21:17:51
✓ returns the most important job if it is in the allowList (2 ms)
21:17:51
✓ returns any job if filter is empty (2 ms)
21:17:51
✓ returns a new job when reporting job success (1 ms)
21:17:51
✓ returns a new job when reporting permanent error (1 ms)
21:17:51
✓ returns a new job when reporting retry-able error (2 ms)
21:17:51
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
21:17:51
✓ returns a new job if job is already in progress elsewhere (1055 ms)
21:17:51
✓ avoids sending the same job to a new agent after a restart (1003 ms)
21:17:51
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
21:17:51
✓ tracks job result if in progress (3 ms)
21:17:51
✓ tracks job result even if job is in queue (2 ms)
21:17:51
✓ ignores reported job error if unknown job (2 ms)
21:17:51
✓ ignores job result if unknown job (1 ms)
21:17:51
✓ returns undefined if no jobs are available (3 ms)
21:17:51
✓ returns jobs in priority order (55 ms)
21:17:51
✓ returns undefined if no jobs are available for the given allowList (25 ms)
21:17:51
✓ returns a job if it is in the allowList (67 ms)
21:17:51
✓ returns the most important job if it is in the allowList (70 ms)
21:17:51
✓ returns any job if filter is empty (65 ms)
21:17:51
✓ returns a new job when reporting job success (51 ms)
21:17:51
✓ returns a new job when reporting permanent error (41 ms)
21:17:51
✓ returns a new job when reporting retry-able error (43 ms)
21:17:51
✓ returns a new job when reporting progress if current one is cancelled (45 ms)
21:17:51
✓ returns a new job if job is already in progress elsewhere (1119 ms)
21:17:51
✓ avoids sending the same job to a new agent after a restart (1050 ms)
21:17:51
✓ avoids sending a completed job to a new agent after a restart (1076 ms)
21:17:51
✓ tracks job result if in progress (67 ms)
21:17:51
✓ tracks job result even if job is in queue (74 ms)
21:17:51
✓ ignores reported job error if unknown job (5 ms)
21:17:51
✓ ignores job result if unknown job (4 ms)
21:17:51 Timeouts
21:17:51
✓ tracks in progress jobs (1 ms)
21:17:51
✓ re-enqueues jobs that time out (109 ms)
21:17:51
✓ cancel stale jobs that time out (236 ms)
21:17:51
✓ keeps the jobs in progress while it is alive (203 ms)
21:17:51
✓ tracks in progress jobs (26 ms)
21:17:51
✓ re-enqueues jobs that time out (134 ms)
21:17:51
✓ cancel stale jobs that time out (329 ms)
21:17:51
✓ keeps the jobs in progress while it is alive (235 ms)
21:17:51 Retries
21:17:51
✓ retries jobs (3 ms)
21:17:51
✓ retries up to a maximum number of times (3 ms)
21:17:51
✓ passing retry=false does not retry (2 ms)
21:17:51
✓ does not retry if job is stale (29 ms)
21:17:51
✓ retries jobs (48 ms)
21:17:51
✓ retries up to a maximum number of times (43 ms)
21:17:51
✓ passing retry=false does not retry (51 ms)
21:17:51
✓ does not retry if job is stale (95 ms)
21:17:51 Database management
21:17:51
✓ re-enqueues proof requests on start (3 ms)
21:17:51
✓ restores proof results on start (1 ms)
21:17:51
✓ only re-enqueues unfinished jobs (2 ms)
21:17:51
✓ saves job when enqueued (3 ms)
21:17:51
✓ does not retain job if database fails to save (3 ms)
21:17:51
✓ saves job result (2 ms)
21:17:51
✓ saves result even if database fails to save (2 ms)
21:17:51
✓ saves job error (2 ms)
21:17:51
✓ saves job error even if database fails to save (3 ms)
21:17:51
✓ does not save job result if job is unknown (2 ms)
21:17:51
✓ does not save job error if job is unknown (1 ms)
21:17:51
✓ cleans up old jobs periodically (261 ms)
21:17:51
✓ re-enqueues proof requests on start (81 ms)
21:17:51
✓ restores proof results on start (76 ms)
21:17:51
✓ only re-enqueues unfinished jobs (78 ms)
21:17:51
✓ saves job when enqueued (32 ms)
21:17:51
✓ does not retain job if database fails to save (6 ms)
21:17:51
✓ saves job result (31 ms)
21:17:51
✓ saves result even if database fails to save (35 ms)
21:17:51
✓ saves job error (49 ms)
21:17:51
✓ saves job error even if database fails to save (37 ms)
21:17:51
✓ does not save job result if job is unknown (4 ms)
21:17:51
✓ does not save job error if job is unknown (4 ms)
21:17:51
✓ cleans up old jobs periodically (399 ms)
21:17:51
21:17:51
Test Suites: 1 passed, 1 total
21:17:51
Tests: 96 passed, 96 total
21:17:51
Snapshots: 0 total
21:17:51
Time: 13.283 s
21:17:51
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
21:17:51
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?