Parent Log:
http://ci.aztec-labs.com/ec1631b4308ef02d
Command: 41a28e9a16c8daa6 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/a9bb2f683797c8e631b892c5f539a2867bddaabf
Env: REF_NAME=gh-readonly-queue/next/pr-15130-e050b787f6dead84c4ef7e5bdb8dcc8cf81d5f65 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Wed Jun 18 15:40:17 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15130_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
15:40:21 [15:40:21.834]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.841]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9430798ed060c96d epochNumber=42
{"provingJobId":"1:BASE_PARITY:9430798ed060c96d"}
15:40:21 [15:40:21.844]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:76041af7dcfb9f9d epochNumber=1
{"provingJobId":"1:BASE_PARITY:76041af7dcfb9f9d"}
15:40:21 [15:40:21.871]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.872]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b60a8f26807cd39 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b60a8f26807cd39"}
15:40:21 [15:40:21.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5586c57fdc71c5af epochNumber=1
{"provingJobId":"1:BASE_PARITY:5586c57fdc71c5af"}
15:40:21 [15:40:21.876]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.877]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d223b764a48e731 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1d223b764a48e731"}
15:40:21 [15:40:21.878]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1d223b764a48e731 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1d223b764a48e731"}
15:40:21 [15:40:21.881]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.881]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2157e81e31a95c6b epochNumber=1
{"provingJobId":"1:BASE_PARITY:2157e81e31a95c6b"}
15:40:21 [15:40:21.883]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2157e81e31a95c6b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2157e81e31a95c6b"}
15:40:21 [15:40:21.883]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2157e81e31a95c6b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2157e81e31a95c6b"}
15:40:21 [15:40:21.884]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2157e81e31a95c6b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2157e81e31a95c6b"}
15:40:21 [15:40:21.886]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.886]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52eba745d0477858 epochNumber=1
{"provingJobId":"1:BASE_PARITY:52eba745d0477858"}
15:40:21 [15:40:21.921]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.923]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.924]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:673331ac5650f9e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:673331ac5650f9e8"}
15:40:21 [15:40:21.925]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:673331ac5650f9e8
{"provingJobId":"1:BASE_PARITY:673331ac5650f9e8"}
15:40:21 [15:40:21.925]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:673331ac5650f9e8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:673331ac5650f9e8"}
15:40:21 [15:40:21.926]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:673331ac5650f9e8 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:673331ac5650f9e8"}
15:40:21 [15:40:21.928]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.928]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f597dba8bd4ed696 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f597dba8bd4ed696"}
15:40:21 [15:40:21.929]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f597dba8bd4ed696
{"provingJobId":"1:BASE_PARITY:f597dba8bd4ed696"}
15:40:21 [15:40:21.929]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f597dba8bd4ed696 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f597dba8bd4ed696"}
15:40:21 [15:40:21.931]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.931]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b509ed2575af6112 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b509ed2575af6112"}
15:40:21 [15:40:21.932]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b509ed2575af6112 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b509ed2575af6112"}
15:40:21 [15:40:21.932]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b509ed2575af6112 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b509ed2575af6112"}
15:40:21 [15:40:21.933]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.933]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a89f43e34f5366e epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a89f43e34f5366e"}
15:40:21 [15:40:21.934]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3a89f43e34f5366e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3a89f43e34f5366e"}
15:40:21 [15:40:21.934]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3a89f43e34f5366e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3a89f43e34f5366e"}
15:40:21 [15:40:21.935]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c185c429d8f410d epochNumber=0
{"provingJobId":"1:BASE_PARITY:7c185c429d8f410d"}
15:40:21 [15:40:21.935]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7c185c429d8f410d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7c185c429d8f410d"}
15:40:21 [15:40:21.936]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a0bfc3c70a8905d epochNumber=0
{"provingJobId":"1:BASE_PARITY:9a0bfc3c70a8905d"}
15:40:21 [15:40:21.936]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7c185c429d8f410d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7c185c429d8f410d"}
15:40:21 [15:40:21.936]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c573067c669a914 epochNumber=0
{"provingJobId":"1:BASE_PARITY:5c573067c669a914"}
15:40:21 [15:40:21.938]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.939]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d5c9e6b3a0c60982 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d5c9e6b3a0c60982"}
15:40:21 [15:40:21.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e91d005ff6d2d42 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e91d005ff6d2d42"}
15:40:21 [15:40:21.940]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.941]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83a480759a76ba67 epochNumber=1
{"provingJobId":"1:BASE_PARITY:83a480759a76ba67"}
15:40:21 [15:40:21.942]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.942]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b7d232b95a67733 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5b7d232b95a67733"}
15:40:21 [15:40:21.942]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b7398957178ee83 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5b7398957178ee83"}
15:40:21 [15:40:21.942]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02849e3aa1e36e0a epochNumber=2
{"provingJobId":"1:BASE_PARITY:02849e3aa1e36e0a"}
15:40:21 [15:40:21.943]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:515f7387f81e4ee4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:515f7387f81e4ee4"}
15:40:21 [15:40:21.943]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3214f6026ca77939 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3214f6026ca77939"}
15:40:21 [15:40:21.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a90dddb07c74e0a1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a90dddb07c74e0a1"}
15:40:21 [15:40:21.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9617a6b41146a9d6 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9617a6b41146a9d6"}
15:40:21 [15:40:21.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:65a48c65172cd8ab epochNumber=1
{"provingJobId":"1:BASE_PARITY:65a48c65172cd8ab"}
15:40:21 [15:40:21.945]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.945]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75d8122268687c21 epochNumber=1
{"provingJobId":"1:BASE_PARITY:75d8122268687c21"}
15:40:21 [15:40:21.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4118395bad6f621c epochNumber=1
{"provingJobId":"1:BASE_PARITY:4118395bad6f621c"}
15:40:21 [15:40:21.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9464cd4877f3553e epochNumber=2
{"provingJobId":"1:BASE_PARITY:9464cd4877f3553e"}
15:40:21 [15:40:21.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4ed971de2ab2c243 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4ed971de2ab2c243"}
15:40:21 [15:40:21.947]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.948]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d1b195977f12649 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d1b195977f12649"}
15:40:21 [15:40:21.949]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0bacb52a62a2b61a epochNumber=1
{"provingJobId":"1:BASE_PARITY:0bacb52a62a2b61a"}
15:40:21 [15:40:21.949]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9d1b195977f12649 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:9d1b195977f12649"}
15:40:21 [15:40:21.951]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.952]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de435b81b19b8712 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de435b81b19b8712"}
15:40:21 [15:40:21.952]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a7734eb1268478df epochNumber=1
{"provingJobId":"1:BASE_PARITY:a7734eb1268478df"}
15:40:21 [15:40:21.953]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:de435b81b19b8712 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:de435b81b19b8712"}
15:40:21 [15:40:21.954]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.954]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90a145dddc1a7844 epochNumber=1
{"provingJobId":"1:BASE_PARITY:90a145dddc1a7844"}
15:40:21 [15:40:21.955]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66c663acb3df9527 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66c663acb3df9527"}
15:40:21 [15:40:21.955]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:90a145dddc1a7844 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:90a145dddc1a7844"}
15:40:21 [15:40:21.956]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.957]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8286ba9687d9ccaf epochNumber=1
{"provingJobId":"1:BASE_PARITY:8286ba9687d9ccaf"}
15:40:21 [15:40:21.957]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:8286ba9687d9ccaf
{"provingJobId":"1:BASE_PARITY:8286ba9687d9ccaf"}
15:40:21 [15:40:21.958]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8286ba9687d9ccaf type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:8286ba9687d9ccaf"}
15:40:21 [15:40:21.958]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd330e5dd8aa1039 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd330e5dd8aa1039"}
15:40:21 [15:40:21.959]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8286ba9687d9ccaf has already been completed
{"provingJobId":"1:BASE_PARITY:8286ba9687d9ccaf"}
15:40:21 [15:40:21.960]
INFO:
prover-client:proving-broker Proving Broker started
15:40:21 [15:40:21.960]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb38696a3b2015f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb38696a3b2015f2"}
15:40:21 [15:40:21.960]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de299467f72448aa epochNumber=2
{"provingJobId":"1:BASE_PARITY:de299467f72448aa"}
15:40:23 [15:40:23.013]
INFO:
prover-client:proving-broker Proving Broker started
15:40:23 [15:40:23.015]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cb38696a3b2015f2 settled=false
{"provingJobId":"1:BASE_PARITY:cb38696a3b2015f2","status":"pending"}
15:40:23 [15:40:23.016]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:de299467f72448aa settled=false
{"provingJobId":"1:BASE_PARITY:de299467f72448aa","status":"pending"}
15:40:23 [15:40:23.017]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cb38696a3b2015f2 type=BASE_PARITY startedAt=1750261221961 older agent has taken job
{"provingJobId":"1:BASE_PARITY:cb38696a3b2015f2"}
15:40:23 [15:40:23.018]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cb38696a3b2015f2 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:cb38696a3b2015f2"}
15:40:23 [15:40:23.019]
INFO:
prover-client:proving-broker Proving Broker started
15:40:23 [15:40:23.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa732e27efd7451f epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa732e27efd7451f"}
15:40:23 [15:40:23.020]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b607053ddde1375 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3b607053ddde1375"}
15:40:24 [15:40:24.022]
INFO:
prover-client:proving-broker Proving Broker started
15:40:24 [15:40:24.022]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fa732e27efd7451f settled=false
{"provingJobId":"1:BASE_PARITY:fa732e27efd7451f","status":"pending"}
15:40:24 [15:40:24.023]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3b607053ddde1375 settled=false
{"provingJobId":"1:BASE_PARITY:3b607053ddde1375","status":"pending"}
15:40:24 [15:40:24.023]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fa732e27efd7451f type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:fa732e27efd7451f"}
15:40:24 [15:40:24.024]
INFO:
prover-client:proving-broker Proving Broker started
15:40:24 [15:40:24.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:534000daa76645cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:534000daa76645cb"}
15:40:24 [15:40:24.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d2b35c789e2e4c8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4d2b35c789e2e4c8"}
15:40:25 [15:40:25.025]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.026]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:534000daa76645cb settled=false
{"provingJobId":"1:BASE_PARITY:534000daa76645cb","status":"pending"}
15:40:25 [15:40:25.026]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4d2b35c789e2e4c8 settled=false
{"provingJobId":"1:BASE_PARITY:4d2b35c789e2e4c8","status":"pending"}
15:40:25 [15:40:25.027]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:534000daa76645cb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:534000daa76645cb"}
15:40:25 [15:40:25.027]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:534000daa76645cb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:534000daa76645cb"}
15:40:25 [15:40:25.028]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:027412f973d5d76b epochNumber=1
{"provingJobId":"1:BASE_PARITY:027412f973d5d76b"}
15:40:25 [15:40:25.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14e4bd07fcc2677c epochNumber=2
{"provingJobId":"1:BASE_PARITY:14e4bd07fcc2677c"}
15:40:25 [15:40:25.030]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:027412f973d5d76b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:027412f973d5d76b"}
15:40:25 [15:40:25.030]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:14e4bd07fcc2677c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:14e4bd07fcc2677c"}
15:40:25 [15:40:25.032]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.032]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa9c05fd5dbca02a epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa9c05fd5dbca02a"}
15:40:25 [15:40:25.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75ae6add59ed2652 epochNumber=2
{"provingJobId":"1:BASE_PARITY:75ae6add59ed2652"}
15:40:25 [15:40:25.033]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fa9c05fd5dbca02a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fa9c05fd5dbca02a"}
15:40:25 [15:40:25.033]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fa9c05fd5dbca02a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fa9c05fd5dbca02a"}
15:40:25 [15:40:25.033]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:75ae6add59ed2652 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:75ae6add59ed2652"}
15:40:25 [15:40:25.034]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.035]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8fb9450ab676d531 err=test error
{"provingJoId":"1:BASE_PARITY:8fb9450ab676d531"}
15:40:25 [15:40:25.036]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.037]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f85f7ca4c87a964d not found
{"provingJobId":"1:BASE_PARITY:f85f7ca4c87a964d"}
15:40:25 [15:40:25.038]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.039]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1778ccd28828c14 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1778ccd28828c14"}
15:40:25 [15:40:25.040]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.040]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6236d30a330e1587 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6236d30a330e1587"}
15:40:25 [15:40:25.142]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6236d30a330e1587 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6236d30a330e1587"}
15:40:25 [15:40:25.150]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.151]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fe6c761336d1e36 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fe6c761336d1e36"}
15:40:25 [15:40:25.276]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4fe6c761336d1e36 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4fe6c761336d1e36"}
15:40:25 [15:40:25.278]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:729e65699498d946 epochNumber=10
{"provingJobId":"1:BASE_PARITY:729e65699498d946"}
15:40:25 [15:40:25.303]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:25 [15:40:25.406]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.407]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f3b61a445d64d65 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f3b61a445d64d65"}
15:40:25 [15:40:25.559]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2f3b61a445d64d65 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2f3b61a445d64d65"}
15:40:25 [15:40:25.610]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.611]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68a9f7631ece7419 epochNumber=1
{"provingJobId":"1:BASE_PARITY:68a9f7631ece7419"}
15:40:25 [15:40:25.612]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:68a9f7631ece7419 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:68a9f7631ece7419"}
15:40:25 [15:40:25.613]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.614]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6675eccf5148ca57 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6675eccf5148ca57"}
15:40:25 [15:40:25.614]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6675eccf5148ca57 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6675eccf5148ca57"}
15:40:25 [15:40:25.615]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6675eccf5148ca57 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:6675eccf5148ca57"}
15:40:25 [15:40:25.616]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.616]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2fdfdce893270142 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2fdfdce893270142"}
15:40:25 [15:40:25.617]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2fdfdce893270142 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:2fdfdce893270142"}
15:40:25 [15:40:25.617]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c63f41cfbd70e018 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c63f41cfbd70e018"}
15:40:25 [15:40:25.618]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c63f41cfbd70e018 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:c63f41cfbd70e018"}
15:40:25 [15:40:25.619]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:23f531dfdceeec46 epochNumber=3
{"provingJobId":"1:BASE_PARITY:23f531dfdceeec46"}
15:40:25 [15:40:25.642]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:25 [15:40:25.644]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c63f41cfbd70e018 err=test error
{"provingJoId":"1:BASE_PARITY:c63f41cfbd70e018"}
15:40:25 [15:40:25.646]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.647]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c7ac1f21a412090d settled=false
{"provingJobId":"1:BASE_PARITY:c7ac1f21a412090d","status":"pending"}
15:40:25 [15:40:25.647]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4cfd27fe26633020 settled=false
{"provingJobId":"1:BASE_PARITY:4cfd27fe26633020","status":"pending"}
15:40:25 [15:40:25.650]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.651]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d4fa0e1dcde15fdb settled=true
{"provingJobId":"1:BASE_PARITY:d4fa0e1dcde15fdb","status":"fulfilled"}
15:40:25 [15:40:25.651]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:3ad9675cfe05d1d4 settled=true
{"provingJobId":"2:BASE_PARITY:3ad9675cfe05d1d4","status":"fulfilled"}
15:40:25 [15:40:25.653]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.653]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7763e4f441f01ef0 settled=true
{"provingJobId":"1:BASE_PARITY:7763e4f441f01ef0","status":"fulfilled"}
15:40:25 [15:40:25.654]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b49616a9ccba3d50 settled=false
{"provingJobId":"1:BASE_PARITY:b49616a9ccba3d50","status":"pending"}
15:40:25 [15:40:25.655]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.656]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb5c5175af8d1bc2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb5c5175af8d1bc2"}
15:40:25 [15:40:25.657]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.658]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e274a102be49c2e epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e274a102be49c2e"}
15:40:25 [15:40:25.658]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:3e274a102be49c2e: Error: db error: Error: db error
15:40:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:3e274a102be49c2e"}
15:40:25 [15:40:25.659]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.660]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2792b99fede4e1a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2792b99fede4e1a4"}
15:40:25 [15:40:25.660]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2792b99fede4e1a4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2792b99fede4e1a4"}
15:40:25 [15:40:25.660]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2792b99fede4e1a4 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2792b99fede4e1a4"}
15:40:25 [15:40:25.661]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.661]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02e7c24687e91bad epochNumber=1
{"provingJobId":"1:BASE_PARITY:02e7c24687e91bad"}
15:40:25 [15:40:25.661]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:02e7c24687e91bad type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:02e7c24687e91bad"}
15:40:25 [15:40:25.662]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:02e7c24687e91bad type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:02e7c24687e91bad"}
15:40:25 [15:40:25.663]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:02e7c24687e91bad: Error: db error
15:40:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:02e7c24687e91bad"}
15:40:25 [15:40:25.665]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.666]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eccfe71d975840fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:eccfe71d975840fe"}
15:40:25 [15:40:25.666]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eccfe71d975840fe type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:eccfe71d975840fe"}
15:40:25 [15:40:25.667]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:eccfe71d975840fe type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:eccfe71d975840fe"}
15:40:25 [15:40:25.668]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.669]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9005d3341a43289c epochNumber=1
{"provingJobId":"1:BASE_PARITY:9005d3341a43289c"}
15:40:25 [15:40:25.670]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9005d3341a43289c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9005d3341a43289c"}
15:40:25 [15:40:25.670]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9005d3341a43289c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9005d3341a43289c"}
15:40:25 [15:40:25.671]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:9005d3341a43289c jobErr=test error: Error: db error
15:40:25 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:9005d3341a43289c"}
15:40:25 [15:40:25.672]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.672]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:495db70f8f1cdaec not found
{"provingJobId":"1:BASE_PARITY:495db70f8f1cdaec"}
15:40:25 [15:40:25.674]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.675]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:49c8e9ac8aa1e42f err=test error
{"provingJoId":"1:BASE_PARITY:49c8e9ac8aa1e42f"}
15:40:25 [15:40:25.676]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.688]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5c9b779bad4b0b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5c9b779bad4b0b8"}
15:40:25 [15:40:25.689]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e5c9b779bad4b0b8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e5c9b779bad4b0b8"}
15:40:25 [15:40:25.689]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e5c9b779bad4b0b8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e5c9b779bad4b0b8"}
15:40:25 [15:40:25.700]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:da095be7f505ca61 epochNumber=2
{"provingJobId":"2:BASE_PARITY:da095be7f505ca61"}
15:40:25 [15:40:25.701]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:da095be7f505ca61 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:da095be7f505ca61"}
15:40:25 [15:40:25.702]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:da095be7f505ca61 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:da095be7f505ca61"}
15:40:25 [15:40:25.763]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:e9394970ac7f4ca1 epochNumber=3
{"provingJobId":"3:BASE_PARITY:e9394970ac7f4ca1"}
15:40:25 [15:40:25.781]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:25 [15:40:25.824]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:e51d79c87c971ab7 epochNumber=4
{"provingJobId":"4:BASE_PARITY:e51d79c87c971ab7"}
15:40:25 [15:40:25.833]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:25 [15:40:25.888]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:2920b33fafe23f47 epochNumber=5
{"provingJobId":"5:BASE_PARITY:2920b33fafe23f47"}
15:40:25 [15:40:25.913]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:25 [15:40:25.943]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3cbe406e744c51f0 epochNumber=42
{"provingJobId":"1:BASE_PARITY:3cbe406e744c51f0"}
15:40:25 [15:40:25.945]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-test7ksO3R/42 with map size 1073741824
15:40:25 [15:40:25.949]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:25 [15:40:25.976]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:1f567dac961cbc9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f567dac961cbc9f"}
15:40:25 [15:40:25.981]
INFO:
prover-client:proving-broker Proving Broker started
15:40:25 [15:40:25.982]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b107a23791dfe06 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b107a23791dfe06"}
15:40:25 [15:40:25.982]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdVPIqh/1 with map size 1073741824
15:40:25 [15:40:25.984]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.010]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7fb69863957e1071 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7fb69863957e1071"}
15:40:26 [15:40:26.018]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.019]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:643ffe28a4412111 epochNumber=1
{"provingJobId":"1:BASE_PARITY:643ffe28a4412111"}
15:40:26 [15:40:26.020]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7h8h12/1 with map size 1073741824
15:40:26 [15:40:26.021]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.044]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:643ffe28a4412111 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:643ffe28a4412111"}
15:40:26 [15:40:26.047]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.047]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b1bf459381130316 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b1bf459381130316"}
15:40:26 [15:40:26.048]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXyhDWY/1 with map size 1073741824
15:40:26 [15:40:26.051]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.069]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b1bf459381130316 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b1bf459381130316"}
15:40:26 [15:40:26.070]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b1bf459381130316 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b1bf459381130316"}
15:40:26 [15:40:26.077]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b1bf459381130316 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b1bf459381130316"}
15:40:26 [15:40:26.080]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.081]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:390533e2eae8a380 epochNumber=1
{"provingJobId":"1:BASE_PARITY:390533e2eae8a380"}
15:40:26 [15:40:26.082]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testV5mfE9/1 with map size 1073741824
15:40:26 [15:40:26.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.117]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.120]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.121]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d0115b819d0b6162 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d0115b819d0b6162"}
15:40:26 [15:40:26.121]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test73N1xc/1 with map size 1073741824
15:40:26 [15:40:26.123]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.141]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d0115b819d0b6162
{"provingJobId":"1:BASE_PARITY:d0115b819d0b6162"}
15:40:26 [15:40:26.142]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d0115b819d0b6162 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d0115b819d0b6162"}
15:40:26 [15:40:26.142]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d0115b819d0b6162 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:d0115b819d0b6162"}
15:40:26 [15:40:26.153]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.154]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f65e27a5713325c epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f65e27a5713325c"}
15:40:26 [15:40:26.154]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSZ0SSc/1 with map size 1073741824
15:40:26 [15:40:26.156]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.174]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5f65e27a5713325c
{"provingJobId":"1:BASE_PARITY:5f65e27a5713325c"}
15:40:26 [15:40:26.174]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5f65e27a5713325c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5f65e27a5713325c"}
15:40:26 [15:40:26.183]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.184]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9e21469b52fd986 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a9e21469b52fd986"}
15:40:26 [15:40:26.185]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCVuj6H/1 with map size 1073741824
15:40:26 [15:40:26.194]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.218]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a9e21469b52fd986 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a9e21469b52fd986"}
15:40:26 [15:40:26.218]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a9e21469b52fd986 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a9e21469b52fd986"}
15:40:26 [15:40:26.227]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.228]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57e34b2d6c0b3a3c epochNumber=1
{"provingJobId":"1:BASE_PARITY:57e34b2d6c0b3a3c"}
15:40:26 [15:40:26.229]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXhOGjw/1 with map size 1073741824
15:40:26 [15:40:26.232]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.252]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:57e34b2d6c0b3a3c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:57e34b2d6c0b3a3c"}
15:40:26 [15:40:26.253]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:57e34b2d6c0b3a3c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:57e34b2d6c0b3a3c"}
15:40:26 [15:40:26.262]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.263]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e1ba65168709314 epochNumber=0
{"provingJobId":"1:BASE_PARITY:3e1ba65168709314"}
15:40:26 [15:40:26.265]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testMpwUnt/0 with map size 1073741824
15:40:26 [15:40:26.268]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.285]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3e1ba65168709314 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3e1ba65168709314"}
15:40:26 [15:40:26.286]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16461f130ee1e45d epochNumber=0
{"provingJobId":"1:BASE_PARITY:16461f130ee1e45d"}
15:40:26 [15:40:26.286]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3e1ba65168709314 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3e1ba65168709314"}
15:40:26 [15:40:26.287]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46979f99c4b918f9 epochNumber=0
{"provingJobId":"1:BASE_PARITY:46979f99c4b918f9"}
15:40:26 [15:40:26.302]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.306]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.306]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4657379001fd3f49 epochNumber=2
{"provingJobId":"1:BASE_PARITY:4657379001fd3f49"}
15:40:26 [15:40:26.307]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDXoiwU/2 with map size 1073741824
15:40:26 [15:40:26.309]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.326]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b5e1253af152856 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b5e1253af152856"}
15:40:26 [15:40:26.327]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDXoiwU/1 with map size 1073741824
15:40:26 [15:40:26.329]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.355]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a3bc1ff1e771d1e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a3bc1ff1e771d1e"}
15:40:26 [15:40:26.358]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0hKWv1/1 with map size 1073741824
15:40:26 [15:40:26.360]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.382]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.384]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd5d00ae3f80b83e epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd5d00ae3f80b83e"}
15:40:26 [15:40:26.387]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2k4nzi/1 with map size 1073741824
15:40:26 [15:40:26.392]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.413]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d1f65513dcd792e epochNumber=1
{"provingJobId":"1:BASE_PARITY:2d1f65513dcd792e"}
15:40:26 [15:40:26.419]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:80956d8cbf9ae5e8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:80956d8cbf9ae5e8"}
15:40:26 [15:40:26.422]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2k4nzi/2 with map size 1073741824
15:40:26 [15:40:26.425]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.444]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:05ed944f3a29c705 epochNumber=1
{"provingJobId":"1:BASE_PARITY:05ed944f3a29c705"}
15:40:26 [15:40:26.454]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.455]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17fcda8dbe3c59de epochNumber=1
{"provingJobId":"1:BASE_PARITY:17fcda8dbe3c59de"}
15:40:26 [15:40:26.456]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGphLe1/1 with map size 1073741824
15:40:26 [15:40:26.458]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.476]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc887a49dda4a650 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bc887a49dda4a650"}
15:40:26 [15:40:26.482]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce99da552cdf7e9a epochNumber=2
{"provingJobId":"1:BASE_PARITY:ce99da552cdf7e9a"}
15:40:26 [15:40:26.483]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGphLe1/2 with map size 1073741824
15:40:26 [15:40:26.488]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7bfab26470ab6c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7bfab26470ab6c8"}
15:40:26 [15:40:26.519]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.519]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:28e8ab8925dfee45 epochNumber=1
{"provingJobId":"1:BASE_PARITY:28e8ab8925dfee45"}
15:40:26 [15:40:26.520]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyxmXyW/1 with map size 1073741824
15:40:26 [15:40:26.522]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.539]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8107226f4bf7b33 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e8107226f4bf7b33"}
15:40:26 [15:40:26.550]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40b42eea068078ef epochNumber=2
{"provingJobId":"1:BASE_PARITY:40b42eea068078ef"}
15:40:26 [15:40:26.551]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyxmXyW/2 with map size 1073741824
15:40:26 [15:40:26.555]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.578]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:551fa43e5b4a93e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:551fa43e5b4a93e6"}
15:40:26 [15:40:26.597]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.599]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:998232b1da2d961e epochNumber=1
{"provingJobId":"1:BASE_PARITY:998232b1da2d961e"}
15:40:26 [15:40:26.601]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4kT2l1/1 with map size 1073741824
15:40:26 [15:40:26.603]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.621]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e92237db91b23bc epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e92237db91b23bc"}
15:40:26 [15:40:26.630]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:998232b1da2d961e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:998232b1da2d961e"}
15:40:26 [15:40:26.641]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.642]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:acd2b17a870c4f36 epochNumber=1
{"provingJobId":"1:BASE_PARITY:acd2b17a870c4f36"}
15:40:26 [15:40:26.642]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWNBOQL/1 with map size 1073741824
15:40:26 [15:40:26.645]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.663]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fa4f4b37af3153b epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fa4f4b37af3153b"}
15:40:26 [15:40:26.672]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:acd2b17a870c4f36 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:acd2b17a870c4f36"}
15:40:26 [15:40:26.683]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.684]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:db32a22888f97817 epochNumber=1
{"provingJobId":"1:BASE_PARITY:db32a22888f97817"}
15:40:26 [15:40:26.685]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsKqphh/1 with map size 1073741824
15:40:26 [15:40:26.692]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.718]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a81e668cb5818059 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a81e668cb5818059"}
15:40:26 [15:40:26.724]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:db32a22888f97817 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:db32a22888f97817"}
15:40:26 [15:40:26.731]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.732]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9597a0632765a5b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9597a0632765a5b1"}
15:40:26 [15:40:26.733]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzI5GcM/1 with map size 1073741824
15:40:26 [15:40:26.735]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.756]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9597a0632765a5b1
{"provingJobId":"1:BASE_PARITY:9597a0632765a5b1"}
15:40:26 [15:40:26.757]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9597a0632765a5b1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:9597a0632765a5b1"}
15:40:26 [15:40:26.763]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b796ab0e93b8a49 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5b796ab0e93b8a49"}
15:40:26 [15:40:26.771]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9597a0632765a5b1 has already been completed
{"provingJobId":"1:BASE_PARITY:9597a0632765a5b1"}
15:40:26 [15:40:26.777]
INFO:
prover-client:proving-broker Proving Broker started
15:40:26 [15:40:26.778]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4e6538fef2c20a4e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4e6538fef2c20a4e"}
15:40:26 [15:40:26.779]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnc7q0c/1 with map size 1073741824
15:40:26 [15:40:26.782]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:26 [15:40:26.801]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b31fe8f0e9808ca epochNumber=2
{"provingJobId":"1:BASE_PARITY:6b31fe8f0e9808ca"}
15:40:26 [15:40:26.803]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnc7q0c/2 with map size 1073741824
15:40:26 [15:40:26.806]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:27 [15:40:27.876]
INFO:
prover-client:proving-broker Proving Broker started
15:40:27 [15:40:27.883]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4e6538fef2c20a4e settled=false
{"provingJobId":"1:BASE_PARITY:4e6538fef2c20a4e","status":"pending"}
15:40:27 [15:40:27.892]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6b31fe8f0e9808ca settled=false
{"provingJobId":"1:BASE_PARITY:6b31fe8f0e9808ca","status":"pending"}
15:40:27 [15:40:27.895]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4e6538fef2c20a4e type=BASE_PARITY startedAt=1750261226824 older agent has taken job
{"provingJobId":"1:BASE_PARITY:4e6538fef2c20a4e"}
15:40:27 [15:40:27.896]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4e6538fef2c20a4e type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:4e6538fef2c20a4e"}
15:40:27 [15:40:27.901]
INFO:
prover-client:proving-broker Proving Broker started
15:40:27 [15:40:27.903]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbb06466f423af63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dbb06466f423af63"}
15:40:27 [15:40:27.904]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt7NKTb/1 with map size 1073741824
15:40:27 [15:40:27.909]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:27 [15:40:27.927]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45afe4c14ec6da13 epochNumber=2
{"provingJobId":"1:BASE_PARITY:45afe4c14ec6da13"}
15:40:27 [15:40:27.928]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testt7NKTb/2 with map size 1073741824
15:40:27 [15:40:27.931]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:28 [15:40:28.948]
INFO:
prover-client:proving-broker Proving Broker started
15:40:28 [15:40:28.951]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dbb06466f423af63 settled=false
{"provingJobId":"1:BASE_PARITY:dbb06466f423af63","status":"pending"}
15:40:28 [15:40:28.953]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:45afe4c14ec6da13 settled=false
{"provingJobId":"1:BASE_PARITY:45afe4c14ec6da13","status":"pending"}
15:40:28 [15:40:28.955]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:dbb06466f423af63 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:dbb06466f423af63"}
15:40:28 [15:40:28.960]
INFO:
prover-client:proving-broker Proving Broker started
15:40:28 [15:40:28.961]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a37574b439a5fd1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a37574b439a5fd1f"}
15:40:28 [15:40:28.961]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCXsEih/1 with map size 1073741824
15:40:28 [15:40:28.963]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:28 [15:40:28.984]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f7d0b7a0a2d4323 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1f7d0b7a0a2d4323"}
15:40:28 [15:40:28.985]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCXsEih/2 with map size 1073741824
15:40:28 [15:40:28.987]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.007]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.009]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a37574b439a5fd1f settled=false
{"provingJobId":"1:BASE_PARITY:a37574b439a5fd1f","status":"pending"}
15:40:30 [15:40:30.012]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1f7d0b7a0a2d4323 settled=false
{"provingJobId":"1:BASE_PARITY:1f7d0b7a0a2d4323","status":"pending"}
15:40:30 [15:40:30.013]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a37574b439a5fd1f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a37574b439a5fd1f"}
15:40:30 [15:40:30.014]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a37574b439a5fd1f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a37574b439a5fd1f"}
15:40:30 [15:40:30.026]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.028]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a3cbc73edd26a3c epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a3cbc73edd26a3c"}
15:40:30 [15:40:30.028]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbnXr2j/1 with map size 1073741824
15:40:30 [15:40:30.032]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.052]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:568c70d908da30cb epochNumber=2
{"provingJobId":"1:BASE_PARITY:568c70d908da30cb"}
15:40:30 [15:40:30.053]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testbnXr2j/2 with map size 1073741824
15:40:30 [15:40:30.057]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.084]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4a3cbc73edd26a3c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4a3cbc73edd26a3c"}
15:40:30 [15:40:30.096]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:568c70d908da30cb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:568c70d908da30cb"}
15:40:30 [15:40:30.109]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.111]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a511ff266e6aceaf epochNumber=1
{"provingJobId":"1:BASE_PARITY:a511ff266e6aceaf"}
15:40:30 [15:40:30.112]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testak0Nro/1 with map size 1073741824
15:40:30 [15:40:30.114]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.132]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55b2b2650dd77a47 epochNumber=2
{"provingJobId":"1:BASE_PARITY:55b2b2650dd77a47"}
15:40:30 [15:40:30.133]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testak0Nro/2 with map size 1073741824
15:40:30 [15:40:30.136]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.154]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a511ff266e6aceaf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a511ff266e6aceaf"}
15:40:30 [15:40:30.155]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a511ff266e6aceaf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a511ff266e6aceaf"}
15:40:30 [15:40:30.162]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:55b2b2650dd77a47 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:55b2b2650dd77a47"}
15:40:30 [15:40:30.173]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.174]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d4054acd08dcab66 err=test error
{"provingJoId":"1:BASE_PARITY:d4054acd08dcab66"}
15:40:30 [15:40:30.179]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.180]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8e9432abe92a2283 not found
{"provingJobId":"1:BASE_PARITY:8e9432abe92a2283"}
15:40:30 [15:40:30.183]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.184]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b446b2641f83f90f epochNumber=1
{"provingJobId":"1:BASE_PARITY:b446b2641f83f90f"}
15:40:30 [15:40:30.187]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyeDpM0/1 with map size 1073741824
15:40:30 [15:40:30.193]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.216]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.217]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b70d66026a957ef7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b70d66026a957ef7"}
15:40:30 [15:40:30.218]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testatNrTu/1 with map size 1073741824
15:40:30 [15:40:30.219]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.344]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b70d66026a957ef7 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b70d66026a957ef7"}
15:40:30 [15:40:30.351]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.352]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:63a84570dda0ce51 epochNumber=1
{"provingJobId":"1:BASE_PARITY:63a84570dda0ce51"}
15:40:30 [15:40:30.353]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4GA0Dt/1 with map size 1073741824
15:40:30 [15:40:30.356]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.505]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:63a84570dda0ce51 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:63a84570dda0ce51"}
15:40:30 [15:40:30.509]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f29f43f33f7ef70 epochNumber=10
{"provingJobId":"1:BASE_PARITY:5f29f43f33f7ef70"}
15:40:30 [15:40:30.510]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test4GA0Dt/10 with map size 1073741824
15:40:30 [15:40:30.513]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.557]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:30 [15:40:30.558]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:40:30 [15:40:30.662]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.663]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99efb648518daf85 epochNumber=1
{"provingJobId":"1:BASE_PARITY:99efb648518daf85"}
15:40:30 [15:40:30.664]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzUZNFD/1 with map size 1073741824
15:40:30 [15:40:30.667]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.847]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:99efb648518daf85 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:99efb648518daf85"}
15:40:30 [15:40:30.902]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.903]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0979b9b93becc768 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0979b9b93becc768"}
15:40:30 [15:40:30.904]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhCsaiM/1 with map size 1073741824
15:40:30 [15:40:30.906]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.928]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0979b9b93becc768 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0979b9b93becc768"}
15:40:30 [15:40:30.932]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.933]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e57c0f1b8d175131 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e57c0f1b8d175131"}
15:40:30 [15:40:30.934]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testumsRap/1 with map size 1073741824
15:40:30 [15:40:30.935]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:30 [15:40:30.960]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e57c0f1b8d175131 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:e57c0f1b8d175131"}
15:40:30 [15:40:30.961]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e57c0f1b8d175131 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:e57c0f1b8d175131"}
15:40:30 [15:40:30.972]
INFO:
prover-client:proving-broker Proving Broker started
15:40:30 [15:40:30.973]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ea7ad3f047ab7be epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ea7ad3f047ab7be"}
15:40:30 [15:40:30.976]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testby2jU3/1 with map size 1073741824
15:40:30 [15:40:30.978]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.003]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6ea7ad3f047ab7be type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6ea7ad3f047ab7be"}
15:40:31 [15:40:31.016]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.017]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83f1c64d3ec8f85f epochNumber=1
{"provingJobId":"1:BASE_PARITY:83f1c64d3ec8f85f"}
15:40:31 [15:40:31.019]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testq9zDfR/1 with map size 1073741824
15:40:31 [15:40:31.022]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.044]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:83f1c64d3ec8f85f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:83f1c64d3ec8f85f"}
15:40:31 [15:40:31.045]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0861353e3d4da7a9 epochNumber=3
{"provingJobId":"1:BASE_PARITY:0861353e3d4da7a9"}
15:40:31 [15:40:31.046]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testq9zDfR/3 with map size 1073741824
15:40:31 [15:40:31.049]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.094]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:31 [15:40:31.094]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:40:31 [15:40:31.095]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:83f1c64d3ec8f85f err=test error
{"provingJoId":"1:BASE_PARITY:83f1c64d3ec8f85f"}
15:40:31 [15:40:31.103]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuG3FUe/1 with map size 1073741824
15:40:31 [15:40:31.106]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.125]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testuG3FUe/2 with map size 1073741824
15:40:31 [15:40:31.127]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.147]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.149]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:845d23f4e11f5131 settled=false
{"provingJobId":"1:BASE_PARITY:845d23f4e11f5131","status":"pending"}
15:40:31 [15:40:31.151]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9d7d067c700c01e0 settled=false
{"provingJobId":"1:BASE_PARITY:9d7d067c700c01e0","status":"pending"}
15:40:31 [15:40:31.159]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrcFb2H/1 with map size 1073741824
15:40:31 [15:40:31.163]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.185]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrcFb2H/2 with map size 1073741824
15:40:31 [15:40:31.189]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.225]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.227]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:add59ace2fca52d4 settled=true
{"provingJobId":"1:BASE_PARITY:add59ace2fca52d4","status":"fulfilled"}
15:40:31 [15:40:31.230]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:a4c3cc7340263cfa settled=true
{"provingJobId":"2:BASE_PARITY:a4c3cc7340263cfa","status":"fulfilled"}
15:40:31 [15:40:31.240]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsdXFiu/1 with map size 1073741824
15:40:31 [15:40:31.243]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.271]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testsdXFiu/2 with map size 1073741824
15:40:31 [15:40:31.274]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.301]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.304]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2f6139ed912bee79 settled=true
{"provingJobId":"1:BASE_PARITY:2f6139ed912bee79","status":"fulfilled"}
15:40:31 [15:40:31.307]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:eb246f88e2508278 settled=false
{"provingJobId":"1:BASE_PARITY:eb246f88e2508278","status":"pending"}
15:40:31 [15:40:31.314]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.315]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a16d4d83ea837153 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a16d4d83ea837153"}
15:40:31 [15:40:31.316]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testR9EW2p/1 with map size 1073741824
15:40:31 [15:40:31.319]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.346]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.347]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:915afe4ffafa2617 epochNumber=1
{"provingJobId":"1:BASE_PARITY:915afe4ffafa2617"}
15:40:31 [15:40:31.348]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:915afe4ffafa2617: Error: db error: Error: db error
15:40:31 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:915afe4ffafa2617"}
15:40:31 [15:40:31.356]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.357]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46228ae1bf0c3fa3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:46228ae1bf0c3fa3"}
15:40:31 [15:40:31.362]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7PwVrJ/1 with map size 1073741824
15:40:31 [15:40:31.373]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.407]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:46228ae1bf0c3fa3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:46228ae1bf0c3fa3"}
15:40:31 [15:40:31.408]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:46228ae1bf0c3fa3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:46228ae1bf0c3fa3"}
15:40:31 [15:40:31.426]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.427]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:257e7e8d03b1e70b epochNumber=1
{"provingJobId":"1:BASE_PARITY:257e7e8d03b1e70b"}
15:40:31 [15:40:31.429]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0azQU8/1 with map size 1073741824
15:40:31 [15:40:31.434]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.468]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:257e7e8d03b1e70b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:257e7e8d03b1e70b"}
15:40:31 [15:40:31.469]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:257e7e8d03b1e70b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:257e7e8d03b1e70b"}
15:40:31 [15:40:31.470]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:257e7e8d03b1e70b: Error: db error
15:40:31 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:257e7e8d03b1e70b"}
15:40:31 [15:40:31.477]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.478]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bbd3f74bddeea383 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bbd3f74bddeea383"}
15:40:31 [15:40:31.479]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmbqozT/1 with map size 1073741824
15:40:31 [15:40:31.483]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.517]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bbd3f74bddeea383 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bbd3f74bddeea383"}
15:40:31 [15:40:31.518]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bbd3f74bddeea383 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bbd3f74bddeea383"}
15:40:31 [15:40:31.537]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.538]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1bb234cf678cc29c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1bb234cf678cc29c"}
15:40:31 [15:40:31.540]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBoQnvF/1 with map size 1073741824
15:40:31 [15:40:31.543]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.568]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1bb234cf678cc29c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1bb234cf678cc29c"}
15:40:31 [15:40:31.569]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1bb234cf678cc29c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1bb234cf678cc29c"}
15:40:31 [15:40:31.570]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:1bb234cf678cc29c jobErr=test error: Error: db error
15:40:31 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:1bb234cf678cc29c"}
15:40:31 [15:40:31.577]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.578]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:87a07d0e36d28bb5 not found
{"provingJobId":"1:BASE_PARITY:87a07d0e36d28bb5"}
15:40:31 [15:40:31.582]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.582]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2db782e7cb010128 err=test error
{"provingJoId":"1:BASE_PARITY:2db782e7cb010128"}
15:40:31 [15:40:31.591]
INFO:
prover-client:proving-broker Proving Broker started
15:40:31 [15:40:31.602]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77e820f29adefcbd epochNumber=1
{"provingJobId":"1:BASE_PARITY:77e820f29adefcbd"}
15:40:31 [15:40:31.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test99HGZ9/1 with map size 1073741824
15:40:31 [15:40:31.610]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.642]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:77e820f29adefcbd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:77e820f29adefcbd"}
15:40:31 [15:40:31.643]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:77e820f29adefcbd type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:77e820f29adefcbd"}
15:40:31 [15:40:31.661]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:666d8b80e5ddf972 epochNumber=2
{"provingJobId":"2:BASE_PARITY:666d8b80e5ddf972"}
15:40:31 [15:40:31.664]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test99HGZ9/2 with map size 1073741824
15:40:31 [15:40:31.669]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.694]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:666d8b80e5ddf972 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:666d8b80e5ddf972"}
15:40:31 [15:40:31.695]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:666d8b80e5ddf972 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:666d8b80e5ddf972"}
15:40:31 [15:40:31.766]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:3571366d0d6bf7b4 epochNumber=3
{"provingJobId":"3:BASE_PARITY:3571366d0d6bf7b4"}
15:40:31 [15:40:31.768]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test99HGZ9/3 with map size 1073741824
15:40:31 [15:40:31.771]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.826]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:31 [15:40:31.827]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
15:40:31 [15:40:31.863]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:b2e067a528d48993 epochNumber=4
{"provingJobId":"4:BASE_PARITY:b2e067a528d48993"}
15:40:31 [15:40:31.864]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-test99HGZ9/4 with map size 1073741824
15:40:31 [15:40:31.866]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.908]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:31 [15:40:31.909]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
15:40:31 [15:40:31.953]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:67b363a3cc6634c9 epochNumber=5
{"provingJobId":"5:BASE_PARITY:67b363a3cc6634c9"}
15:40:31 [15:40:31.955]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-test99HGZ9/5 with map size 1073741824
15:40:31 [15:40:31.958]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
15:40:31 [15:40:31.988]
INFO:
prover-client:proving-broker Cleaned up jobs=1
15:40:31 [15:40:31.991]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
15:40:32
PASS src/proving_broker/proving_broker.test.ts (
13.673 s)
15:40:32 ProvingBroker
15:40:32 Producer API
15:40:32
✓ refuses stale jobs (39 ms)
15:40:32
✓ enqueues jobs (4 ms)
15:40:32
✓ ignores duplicate jobs (5 ms)
15:40:32
✓ reports correct status when enqueuing repeat jobs (5 ms)
15:40:32
✓ throws an error in case of duplicate job IDs (35 ms)
15:40:32
✓ returns not-found status for non-existing jobs (1 ms)
15:40:32
✓ cancels jobs in queue (5 ms)
15:40:32
✓ cancels jobs in-progress (3 ms)
15:40:32
✓ returns job result if successful (2 ms)
15:40:32
✓ returns job error if failed (1 ms)
15:40:32
✓ correctly returns job status for concurrent writes (3 ms)
15:40:32
✓ refuses stale jobs (38 ms)
15:40:32
✓ enqueues jobs (38 ms)
15:40:32
✓ ignores duplicate jobs (28 ms)
15:40:32
✓ reports correct status when enqueuing repeat jobs (33 ms)
15:40:32
✓ throws an error in case of duplicate job IDs (36 ms)
15:40:32
✓ returns not-found status for non-existing jobs (3 ms)
15:40:32
✓ cancels jobs in queue (32 ms)
15:40:32
✓ cancels jobs in-progress (30 ms)
15:40:32
✓ returns job result if successful (43 ms)
15:40:32
✓ returns job error if failed (35 ms)
15:40:32
✓ correctly returns job status for concurrent writes (40 ms)
15:40:32 Consumer API
15:40:32
✓ returns undefined if no jobs are available (1 ms)
15:40:32
✓ returns jobs in priority order (1 ms)
15:40:32
✓ returns undefined if no jobs are available for the given allowList (1 ms)
15:40:32
✓ returns a job if it is in the allowList (2 ms)
15:40:32
✓ returns the most important job if it is in the allowList (2 ms)
15:40:32
✓ returns any job if filter is empty (2 ms)
15:40:32
✓ returns a new job when reporting job success (3 ms)
15:40:32
✓ returns a new job when reporting permanent error (2 ms)
15:40:32
✓ returns a new job when reporting retry-able error (2 ms)
15:40:32
✓ returns a new job when reporting progress if current one is cancelled (4 ms)
15:40:32
✓ returns a new job if job is already in progress elsewhere (1060 ms)
15:40:32
✓ avoids sending the same job to a new agent after a restart (1005 ms)
15:40:32
✓ avoids sending a completed job to a new agent after a restart (1004 ms)
15:40:32
✓ tracks job result if in progress (3 ms)
15:40:32
✓ tracks job result even if job is in queue (3 ms)
15:40:32
✓ ignores reported job error if unknown job (1 ms)
15:40:32
✓ ignores job result if unknown job (1 ms)
15:40:32
✓ returns undefined if no jobs are available (3 ms)
15:40:32
✓ returns jobs in priority order (50 ms)
15:40:32
✓ returns undefined if no jobs are available for the given allowList (26 ms)
15:40:32
✓ returns a job if it is in the allowList (72 ms)
15:40:32
✓ returns the most important job if it is in the allowList (64 ms)
15:40:32
✓ returns any job if filter is empty (74 ms)
15:40:32
✓ returns a new job when reporting job success (46 ms)
15:40:32
✓ returns a new job when reporting permanent error (42 ms)
15:40:32
✓ returns a new job when reporting retry-able error (45 ms)
15:40:32
✓ returns a new job when reporting progress if current one is cancelled (47 ms)
15:40:32
✓ returns a new job if job is already in progress elsewhere (1125 ms)
15:40:32
✓ avoids sending the same job to a new agent after a restart (1058 ms)
15:40:32
✓ avoids sending a completed job to a new agent after a restart (1065 ms)
15:40:32
✓ tracks job result if in progress (83 ms)
15:40:32
✓ tracks job result even if job is in queue (63 ms)
15:40:32
✓ ignores reported job error if unknown job (3 ms)
15:40:32
✓ ignores job result if unknown job (6 ms)
15:40:32 Timeouts
15:40:32
✓ tracks in progress jobs (1 ms)
15:40:32
✓ re-enqueues jobs that time out (109 ms)
15:40:32
✓ cancel stale jobs that time out (256 ms)
15:40:32
✓ keeps the jobs in progress while it is alive (204 ms)
15:40:32
✓ tracks in progress jobs (33 ms)
15:40:32
✓ re-enqueues jobs that time out (132 ms)
15:40:32
✓ cancel stale jobs that time out (312 ms)
15:40:32
✓ keeps the jobs in progress while it is alive (239 ms)
15:40:32 Retries
15:40:32
✓ retries jobs (2 ms)
15:40:32
✓ retries up to a maximum number of times (2 ms)
15:40:32
✓ passing retry=false does not retry (2 ms)
15:40:32
✓ does not retry if job is stale (28 ms)
15:40:32
✓ retries jobs (30 ms)
15:40:32
✓ retries up to a maximum number of times (40 ms)
15:40:32
✓ passing retry=false does not retry (41 ms)
15:40:32
✓ does not retry if job is stale (85 ms)
15:40:32 Database management
15:40:32
✓ re-enqueues proof requests on start (4 ms)
15:40:32
✓ restores proof results on start (2 ms)
15:40:32
✓ only re-enqueues unfinished jobs (2 ms)
15:40:32
✓ saves job when enqueued (2 ms)
15:40:32
✓ does not retain job if database fails to save (2 ms)
15:40:32
✓ saves job result (2 ms)
15:40:32
✓ saves result even if database fails to save (3 ms)
15:40:32
✓ saves job error (4 ms)
15:40:32
✓ saves job error even if database fails to save (3 ms)
15:40:32
✓ does not save job result if job is unknown (2 ms)
15:40:32
✓ does not save job error if job is unknown (1 ms)
15:40:32
✓ cleans up old jobs periodically (265 ms)
15:40:32
✓ re-enqueues proof requests on start (57 ms)
15:40:32
✓ restores proof results on start (78 ms)
15:40:32
✓ only re-enqueues unfinished jobs (76 ms)
15:40:32
✓ saves job when enqueued (33 ms)
15:40:32
✓ does not retain job if database fails to save (7 ms)
15:40:32
✓ saves job result (69 ms)
15:40:32
✓ saves result even if database fails to save (51 ms)
15:40:32
✓ saves job error (57 ms)
15:40:32
✓ saves job error even if database fails to save (43 ms)
15:40:32
✓ does not save job result if job is unknown (5 ms)
15:40:32
✓ does not save job error if job is unknown (5 ms)
15:40:32
✓ cleans up old jobs periodically (445 ms)
15:40:32
15:40:32
Test Suites: 1 passed, 1 total
15:40:32
Tests: 96 passed, 96 total
15:40:32
Snapshots: 0 total
15:40:32
Time: 13.776 s
15:40:32
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
15:40:32
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?