Parent Log:
http://ci.aztec-labs.com/da84e1400af72320
Command: 84c999e211bb73e4 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/3863bc02d671057d0cffd4ced17ca317788ff190
Env: REF_NAME=gh-readonly-queue/staging/pr-15238-db396f13b7ef374a230174bdc4078d9fbb3c788a CURRENT_VERSION=1.0.0 CI_FULL=1
Date: Tue Jun 24 12:12:17 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15238_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_staging
12:12:22 [12:12:22.583]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.590]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:981ef0ac50d46f70 epochNumber=42
{"provingJobId":"1:BASE_PARITY:981ef0ac50d46f70"}
12:12:22 [12:12:22.593]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:71fbfb66e49f5e62 epochNumber=1
{"provingJobId":"1:BASE_PARITY:71fbfb66e49f5e62"}
12:12:22 [12:12:22.628]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.629]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7216dcb5bbe68d64 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7216dcb5bbe68d64"}
12:12:22 [12:12:22.631]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2245d7236f4c9e69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2245d7236f4c9e69"}
12:12:22 [12:12:22.634]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.634]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02f134464ef81121 epochNumber=1
{"provingJobId":"1:BASE_PARITY:02f134464ef81121"}
12:12:22 [12:12:22.635]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:02f134464ef81121 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:02f134464ef81121"}
12:12:22 [12:12:22.638]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.638]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12a0f29e383bf1be epochNumber=1
{"provingJobId":"1:BASE_PARITY:12a0f29e383bf1be"}
12:12:22 [12:12:22.640]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:12a0f29e383bf1be epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:12a0f29e383bf1be"}
12:12:22 [12:12:22.640]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:12a0f29e383bf1be type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:12a0f29e383bf1be"}
12:12:22 [12:12:22.641]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:12a0f29e383bf1be epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:12a0f29e383bf1be"}
12:12:22 [12:12:22.642]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.643]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04ae583055dfb4a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:04ae583055dfb4a0"}
12:12:22 [12:12:22.669]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.670]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.671]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a43b49bc26d7f5fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:a43b49bc26d7f5fe"}
12:12:22 [12:12:22.671]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a43b49bc26d7f5fe
{"provingJobId":"1:BASE_PARITY:a43b49bc26d7f5fe"}
12:12:22 [12:12:22.672]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a43b49bc26d7f5fe type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a43b49bc26d7f5fe"}
12:12:22 [12:12:22.672]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a43b49bc26d7f5fe type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:a43b49bc26d7f5fe"}
12:12:22 [12:12:22.673]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.674]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96d885123d85391d epochNumber=1
{"provingJobId":"1:BASE_PARITY:96d885123d85391d"}
12:12:22 [12:12:22.675]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:96d885123d85391d
{"provingJobId":"1:BASE_PARITY:96d885123d85391d"}
12:12:22 [12:12:22.675]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:96d885123d85391d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:96d885123d85391d"}
12:12:22 [12:12:22.676]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.677]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6fce073a41e50a8c epochNumber=1
{"provingJobId":"1:BASE_PARITY:6fce073a41e50a8c"}
12:12:22 [12:12:22.677]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6fce073a41e50a8c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6fce073a41e50a8c"}
12:12:22 [12:12:22.677]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6fce073a41e50a8c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6fce073a41e50a8c"}
12:12:22 [12:12:22.679]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.679]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3276f06946a003ef epochNumber=1
{"provingJobId":"1:BASE_PARITY:3276f06946a003ef"}
12:12:22 [12:12:22.679]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3276f06946a003ef type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3276f06946a003ef"}
12:12:22 [12:12:22.680]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3276f06946a003ef type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3276f06946a003ef"}
12:12:22 [12:12:22.681]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.681]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34f7d69e58362a48 epochNumber=0
{"provingJobId":"1:BASE_PARITY:34f7d69e58362a48"}
12:12:22 [12:12:22.681]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:34f7d69e58362a48 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:34f7d69e58362a48"}
12:12:22 [12:12:22.682]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b5b8202dccbc4f6 epochNumber=0
{"provingJobId":"1:BASE_PARITY:8b5b8202dccbc4f6"}
12:12:22 [12:12:22.682]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:34f7d69e58362a48 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:34f7d69e58362a48"}
12:12:22 [12:12:22.682]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0be60b9cf23d6043 epochNumber=0
{"provingJobId":"1:BASE_PARITY:0be60b9cf23d6043"}
12:12:22 [12:12:22.693]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.695]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.697]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:908000b801fb1cb9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:908000b801fb1cb9"}
12:12:22 [12:12:22.697]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f0c35ce60cabe5e epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f0c35ce60cabe5e"}
12:12:22 [12:12:22.699]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d00352e4905df1e7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d00352e4905df1e7"}
12:12:22 [12:12:22.702]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.702]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25b2c3c12eb4e405 epochNumber=1
{"provingJobId":"1:BASE_PARITY:25b2c3c12eb4e405"}
12:12:22 [12:12:22.703]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aeaf51da64164dea epochNumber=1
{"provingJobId":"1:BASE_PARITY:aeaf51da64164dea"}
12:12:22 [12:12:22.703]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72ffcf2abdb096a1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:72ffcf2abdb096a1"}
12:12:22 [12:12:22.704]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5af6bafbabd77119 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5af6bafbabd77119"}
12:12:22 [12:12:22.705]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.706]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ffa3f4a58f09a27 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1ffa3f4a58f09a27"}
12:12:22 [12:12:22.706]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f289f30f9f8a1bc7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f289f30f9f8a1bc7"}
12:12:22 [12:12:22.706]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9bd9e9a6695d6550 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9bd9e9a6695d6550"}
12:12:22 [12:12:22.707]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9371930186e4632d epochNumber=1
{"provingJobId":"1:BASE_PARITY:9371930186e4632d"}
12:12:22 [12:12:22.709]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.710]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:896ab8cdcace1e4c epochNumber=1
{"provingJobId":"1:BASE_PARITY:896ab8cdcace1e4c"}
12:12:22 [12:12:22.710]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fb9d719af917ee5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fb9d719af917ee5"}
12:12:22 [12:12:22.711]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfbb3c4a0f335bb7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dfbb3c4a0f335bb7"}
12:12:22 [12:12:22.711]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57f64baacd6caa3f epochNumber=1
{"provingJobId":"1:BASE_PARITY:57f64baacd6caa3f"}
12:12:22 [12:12:22.713]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.713]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a2208ea7c2538203 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a2208ea7c2538203"}
12:12:22 [12:12:22.714]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:759612be39201a87 epochNumber=1
{"provingJobId":"1:BASE_PARITY:759612be39201a87"}
12:12:22 [12:12:22.714]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a2208ea7c2538203 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a2208ea7c2538203"}
12:12:22 [12:12:22.716]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a979812a4d24dfe epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a979812a4d24dfe"}
12:12:22 [12:12:22.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f5bdb1738b17677a epochNumber=1
{"provingJobId":"1:BASE_PARITY:f5bdb1738b17677a"}
12:12:22 [12:12:22.718]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4a979812a4d24dfe type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:4a979812a4d24dfe"}
12:12:22 [12:12:22.719]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.720]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:998dc7ec335d5707 epochNumber=1
{"provingJobId":"1:BASE_PARITY:998dc7ec335d5707"}
12:12:22 [12:12:22.720]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87e3f5ad9414e799 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87e3f5ad9414e799"}
12:12:22 [12:12:22.720]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:998dc7ec335d5707 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:998dc7ec335d5707"}
12:12:22 [12:12:22.722]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.722]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02cff2175b3b239c epochNumber=1
{"provingJobId":"1:BASE_PARITY:02cff2175b3b239c"}
12:12:22 [12:12:22.723]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:02cff2175b3b239c
{"provingJobId":"1:BASE_PARITY:02cff2175b3b239c"}
12:12:22 [12:12:22.723]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:02cff2175b3b239c type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:02cff2175b3b239c"}
12:12:22 [12:12:22.724]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9549dd08614df7c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9549dd08614df7c3"}
12:12:22 [12:12:22.725]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:02cff2175b3b239c has already been completed
{"provingJobId":"1:BASE_PARITY:02cff2175b3b239c"}
12:12:22 [12:12:22.726]
INFO:
prover-client:proving-broker Proving Broker started
12:12:22 [12:12:22.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ad56426e0277fd8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ad56426e0277fd8"}
12:12:22 [12:12:22.727]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a07b30b0301d852 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7a07b30b0301d852"}
12:12:23 [12:12:23.780]
INFO:
prover-client:proving-broker Proving Broker started
12:12:23 [12:12:23.781]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5ad56426e0277fd8 settled=false
{"provingJobId":"1:BASE_PARITY:5ad56426e0277fd8","status":"pending"}
12:12:23 [12:12:23.782]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7a07b30b0301d852 settled=false
{"provingJobId":"1:BASE_PARITY:7a07b30b0301d852","status":"pending"}
12:12:23 [12:12:23.783]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5ad56426e0277fd8 type=BASE_PARITY startedAt=1750767142728 older agent has taken job
{"provingJobId":"1:BASE_PARITY:5ad56426e0277fd8"}
12:12:23 [12:12:23.784]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5ad56426e0277fd8 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:5ad56426e0277fd8"}
12:12:23 [12:12:23.785]
INFO:
prover-client:proving-broker Proving Broker started
12:12:23 [12:12:23.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:693873c7d9c8fb9b epochNumber=1
{"provingJobId":"1:BASE_PARITY:693873c7d9c8fb9b"}
12:12:23 [12:12:23.786]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c8eb033262602c13 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c8eb033262602c13"}
12:12:24 [12:12:24.788]
INFO:
prover-client:proving-broker Proving Broker started
12:12:24 [12:12:24.788]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:693873c7d9c8fb9b settled=false
{"provingJobId":"1:BASE_PARITY:693873c7d9c8fb9b","status":"pending"}
12:12:24 [12:12:24.788]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c8eb033262602c13 settled=false
{"provingJobId":"1:BASE_PARITY:c8eb033262602c13","status":"pending"}
12:12:24 [12:12:24.789]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:693873c7d9c8fb9b type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:693873c7d9c8fb9b"}
12:12:24 [12:12:24.791]
INFO:
prover-client:proving-broker Proving Broker started
12:12:24 [12:12:24.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a4468be654c2bd9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a4468be654c2bd9"}
12:12:24 [12:12:24.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:abd885c13f3109f7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:abd885c13f3109f7"}
12:12:25 [12:12:25.792]
INFO:
prover-client:proving-broker Proving Broker started
12:12:25 [12:12:25.793]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7a4468be654c2bd9 settled=false
{"provingJobId":"1:BASE_PARITY:7a4468be654c2bd9","status":"pending"}
12:12:25 [12:12:25.794]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:abd885c13f3109f7 settled=false
{"provingJobId":"1:BASE_PARITY:abd885c13f3109f7","status":"pending"}
12:12:25 [12:12:25.795]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7a4468be654c2bd9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7a4468be654c2bd9"}
12:12:25 [12:12:25.795]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7a4468be654c2bd9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7a4468be654c2bd9"}
12:12:25 [12:12:25.797]
INFO:
prover-client:proving-broker Proving Broker started
12:12:25 [12:12:25.798]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e37f5286ef8eac64 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e37f5286ef8eac64"}
12:12:25 [12:12:25.798]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ac32295bec96d20 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3ac32295bec96d20"}
12:12:25 [12:12:25.799]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e37f5286ef8eac64 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e37f5286ef8eac64"}
12:12:25 [12:12:25.799]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3ac32295bec96d20 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3ac32295bec96d20"}
12:12:25 [12:12:25.801]
INFO:
prover-client:proving-broker Proving Broker started
12:12:25 [12:12:25.801]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:589fdac37124985c epochNumber=1
{"provingJobId":"1:BASE_PARITY:589fdac37124985c"}
12:12:25 [12:12:25.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:158c9081b4a20c0e epochNumber=2
{"provingJobId":"1:BASE_PARITY:158c9081b4a20c0e"}
12:12:25 [12:12:25.802]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:589fdac37124985c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:589fdac37124985c"}
12:12:25 [12:12:25.802]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:589fdac37124985c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:589fdac37124985c"}
12:12:25 [12:12:25.803]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:158c9081b4a20c0e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:158c9081b4a20c0e"}
12:12:25 [12:12:25.804]
INFO:
prover-client:proving-broker Proving Broker started
12:12:25 [12:12:25.805]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a34cc6af44ff152d err=test error
{"provingJoId":"1:BASE_PARITY:a34cc6af44ff152d"}
12:12:25 [12:12:25.806]
INFO:
prover-client:proving-broker Proving Broker started
12:12:25 [12:12:25.806]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5c5ea4514d4ce561 not found
{"provingJobId":"1:BASE_PARITY:5c5ea4514d4ce561"}
12:12:25 [12:12:25.808]
INFO:
prover-client:proving-broker Proving Broker started
12:12:25 [12:12:25.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:580eb14a7653babd epochNumber=1
{"provingJobId":"1:BASE_PARITY:580eb14a7653babd"}
12:12:25 [12:12:25.809]
INFO:
prover-client:proving-broker Proving Broker started
12:12:25 [12:12:25.810]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:69eb958b032219c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:69eb958b032219c9"}
12:12:25 [12:12:25.911]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:69eb958b032219c9 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:69eb958b032219c9"}
12:12:25 [12:12:25.919]
INFO:
prover-client:proving-broker Proving Broker started
12:12:25 [12:12:25.920]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60c4cf484b73a3e5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:60c4cf484b73a3e5"}
12:12:26 [12:12:26.021]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:60c4cf484b73a3e5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:60c4cf484b73a3e5"}
12:12:26 [12:12:26.028]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d2d375ae92d4c29 epochNumber=10
{"provingJobId":"1:BASE_PARITY:4d2d375ae92d4c29"}
12:12:26 [12:12:26.047]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:26 [12:12:26.159]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.160]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:362084b1afa54770 epochNumber=1
{"provingJobId":"1:BASE_PARITY:362084b1afa54770"}
12:12:26 [12:12:26.311]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:362084b1afa54770 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:362084b1afa54770"}
12:12:26 [12:12:26.363]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.364]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2bb69b94f8b3cf7f epochNumber=1
{"provingJobId":"1:BASE_PARITY:2bb69b94f8b3cf7f"}
12:12:26 [12:12:26.364]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2bb69b94f8b3cf7f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:2bb69b94f8b3cf7f"}
12:12:26 [12:12:26.366]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bcaca3d0746e9256 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bcaca3d0746e9256"}
12:12:26 [12:12:26.367]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bcaca3d0746e9256 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:bcaca3d0746e9256"}
12:12:26 [12:12:26.367]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bcaca3d0746e9256 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:bcaca3d0746e9256"}
12:12:26 [12:12:26.368]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.368]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc0c4adc8f1b3a93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc0c4adc8f1b3a93"}
12:12:26 [12:12:26.369]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fc0c4adc8f1b3a93 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fc0c4adc8f1b3a93"}
12:12:26 [12:12:26.369]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.370]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf9f8e8aa4ae37bb epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf9f8e8aa4ae37bb"}
12:12:26 [12:12:26.370]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bf9f8e8aa4ae37bb type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:bf9f8e8aa4ae37bb"}
12:12:26 [12:12:26.371]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d2262d1ba05f25f epochNumber=3
{"provingJobId":"1:BASE_PARITY:6d2262d1ba05f25f"}
12:12:26 [12:12:26.395]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:26 [12:12:26.396]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bf9f8e8aa4ae37bb err=test error
{"provingJoId":"1:BASE_PARITY:bf9f8e8aa4ae37bb"}
12:12:26 [12:12:26.406]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.407]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5fe0929740af79c8 settled=false
{"provingJobId":"1:BASE_PARITY:5fe0929740af79c8","status":"pending"}
12:12:26 [12:12:26.407]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:197c31469a233e91 settled=false
{"provingJobId":"1:BASE_PARITY:197c31469a233e91","status":"pending"}
12:12:26 [12:12:26.411]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.412]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:58ead2bd3c984a05 settled=true
{"provingJobId":"1:BASE_PARITY:58ead2bd3c984a05","status":"fulfilled"}
12:12:26 [12:12:26.412]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:e623befb506153fa settled=true
{"provingJobId":"2:BASE_PARITY:e623befb506153fa","status":"fulfilled"}
12:12:26 [12:12:26.414]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.415]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ced0d128e8c67009 settled=true
{"provingJobId":"1:BASE_PARITY:ced0d128e8c67009","status":"fulfilled"}
12:12:26 [12:12:26.415]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cbcdb6c6de841dcd settled=false
{"provingJobId":"1:BASE_PARITY:cbcdb6c6de841dcd","status":"pending"}
12:12:26 [12:12:26.418]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.420]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:408a68a7a8fe2fd7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:408a68a7a8fe2fd7"}
12:12:26 [12:12:26.422]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.423]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:26e6d9a94d5c78bb epochNumber=1
{"provingJobId":"1:BASE_PARITY:26e6d9a94d5c78bb"}
12:12:26 [12:12:26.424]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:26e6d9a94d5c78bb: Error: db error: Error: db error
12:12:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:26e6d9a94d5c78bb"}
12:12:26 [12:12:26.426]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.427]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe22538b73b342aa epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe22538b73b342aa"}
12:12:26 [12:12:26.427]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fe22538b73b342aa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fe22538b73b342aa"}
12:12:26 [12:12:26.427]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fe22538b73b342aa type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fe22538b73b342aa"}
12:12:26 [12:12:26.429]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.429]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0cf36c3367d61038 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0cf36c3367d61038"}
12:12:26 [12:12:26.430]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0cf36c3367d61038 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0cf36c3367d61038"}
12:12:26 [12:12:26.430]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0cf36c3367d61038 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0cf36c3367d61038"}
12:12:26 [12:12:26.431]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:0cf36c3367d61038: Error: db error
12:12:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:0cf36c3367d61038"}
12:12:26 [12:12:26.432]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.433]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:01435cdf487883d4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:01435cdf487883d4"}
12:12:26 [12:12:26.433]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:01435cdf487883d4 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:01435cdf487883d4"}
12:12:26 [12:12:26.434]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:01435cdf487883d4 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:01435cdf487883d4"}
12:12:26 [12:12:26.435]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.436]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bfdb9955d2e4e284 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bfdb9955d2e4e284"}
12:12:26 [12:12:26.436]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bfdb9955d2e4e284 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bfdb9955d2e4e284"}
12:12:26 [12:12:26.436]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bfdb9955d2e4e284 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bfdb9955d2e4e284"}
12:12:26 [12:12:26.437]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:bfdb9955d2e4e284 jobErr=test error: Error: db error
12:12:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:bfdb9955d2e4e284"}
12:12:26 [12:12:26.438]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.439]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:886ae8dd84e85cd5 not found
{"provingJobId":"1:BASE_PARITY:886ae8dd84e85cd5"}
12:12:26 [12:12:26.440]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.441]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b5d08df46b1415e3 err=test error
{"provingJoId":"1:BASE_PARITY:b5d08df46b1415e3"}
12:12:26 [12:12:26.442]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.452]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb47162ed054356c epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb47162ed054356c"}
12:12:26 [12:12:26.454]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb47162ed054356c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:eb47162ed054356c"}
12:12:26 [12:12:26.454]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:eb47162ed054356c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:eb47162ed054356c"}
12:12:26 [12:12:26.465]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:498475da11fd0526 epochNumber=2
{"provingJobId":"2:BASE_PARITY:498475da11fd0526"}
12:12:26 [12:12:26.467]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:498475da11fd0526 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:498475da11fd0526"}
12:12:26 [12:12:26.470]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:498475da11fd0526 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:498475da11fd0526"}
12:12:26 [12:12:26.533]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:01daa6c912f70eec epochNumber=3
{"provingJobId":"3:BASE_PARITY:01daa6c912f70eec"}
12:12:26 [12:12:26.553]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:26 [12:12:26.595]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:bd1925a9988b6989 epochNumber=4
{"provingJobId":"4:BASE_PARITY:bd1925a9988b6989"}
12:12:26 [12:12:26.605]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:26 [12:12:26.659]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:fad181ac8ad6986a epochNumber=5
{"provingJobId":"5:BASE_PARITY:fad181ac8ad6986a"}
12:12:26 [12:12:26.666]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:26 [12:12:26.721]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.722]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:383229f7b1f016af epochNumber=42
{"provingJobId":"1:BASE_PARITY:383229f7b1f016af"}
12:12:26 [12:12:26.724]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testzeIVI3/42 with map size 1073741824
12:12:26 [12:12:26.729]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:26 [12:12:26.762]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:7dccbd35980c7abe epochNumber=1
{"provingJobId":"1:BASE_PARITY:7dccbd35980c7abe"}
12:12:26 [12:12:26.778]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.779]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc3e34cbbd18f81c epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc3e34cbbd18f81c"}
12:12:26 [12:12:26.780]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfXHJo4/1 with map size 1073741824
12:12:26 [12:12:26.783]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:26 [12:12:26.803]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d30c1389a6ca356 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d30c1389a6ca356"}
12:12:26 [12:12:26.817]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.818]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e438906e7c58012e epochNumber=1
{"provingJobId":"1:BASE_PARITY:e438906e7c58012e"}
12:12:26 [12:12:26.819]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmhegmS/1 with map size 1073741824
12:12:26 [12:12:26.822]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:26 [12:12:26.840]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e438906e7c58012e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e438906e7c58012e"}
12:12:26 [12:12:26.845]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ebb0c921384e98d2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ebb0c921384e98d2"}
12:12:26 [12:12:26.847]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCpuWL2/1 with map size 1073741824
12:12:26 [12:12:26.850]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:26 [12:12:26.876]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ebb0c921384e98d2 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ebb0c921384e98d2"}
12:12:26 [12:12:26.878]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ebb0c921384e98d2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ebb0c921384e98d2"}
12:12:26 [12:12:26.886]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ebb0c921384e98d2 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ebb0c921384e98d2"}
12:12:26 [12:12:26.893]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.894]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bdf1f5ed3795f8eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:bdf1f5ed3795f8eb"}
12:12:26 [12:12:26.894]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLzrO6l/1 with map size 1073741824
12:12:26 [12:12:26.897]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:26 [12:12:26.918]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.923]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.924]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95065fb3c5bdefb9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:95065fb3c5bdefb9"}
12:12:26 [12:12:26.925]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2UmN6e/1 with map size 1073741824
12:12:26 [12:12:26.927]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:26 [12:12:26.949]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:95065fb3c5bdefb9
{"provingJobId":"1:BASE_PARITY:95065fb3c5bdefb9"}
12:12:26 [12:12:26.950]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:95065fb3c5bdefb9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:95065fb3c5bdefb9"}
12:12:26 [12:12:26.950]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:95065fb3c5bdefb9 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:95065fb3c5bdefb9"}
12:12:26 [12:12:26.973]
INFO:
prover-client:proving-broker Proving Broker started
12:12:26 [12:12:26.975]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46cdc091c7b4ea56 epochNumber=1
{"provingJobId":"1:BASE_PARITY:46cdc091c7b4ea56"}
12:12:26 [12:12:26.976]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyswASy/1 with map size 1073741824
12:12:26 [12:12:26.981]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.000]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:46cdc091c7b4ea56
{"provingJobId":"1:BASE_PARITY:46cdc091c7b4ea56"}
12:12:27 [12:12:27.001]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:46cdc091c7b4ea56 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:46cdc091c7b4ea56"}
12:12:27 [12:12:27.012]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.013]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce835fdc9fe5ed77 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ce835fdc9fe5ed77"}
12:12:27 [12:12:27.019]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testr83KrO/1 with map size 1073741824
12:12:27 [12:12:27.021]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.041]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ce835fdc9fe5ed77 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ce835fdc9fe5ed77"}
12:12:27 [12:12:27.042]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ce835fdc9fe5ed77 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ce835fdc9fe5ed77"}
12:12:27 [12:12:27.053]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.053]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a5010e7b9480e90 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a5010e7b9480e90"}
12:12:27 [12:12:27.057]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaEatW0/1 with map size 1073741824
12:12:27 [12:12:27.068]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.096]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1a5010e7b9480e90 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1a5010e7b9480e90"}
12:12:27 [12:12:27.096]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1a5010e7b9480e90 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1a5010e7b9480e90"}
12:12:27 [12:12:27.109]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.111]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bba871d020b1250b epochNumber=0
{"provingJobId":"1:BASE_PARITY:bba871d020b1250b"}
12:12:27 [12:12:27.113]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testSEJi5d/0 with map size 1073741824
12:12:27 [12:12:27.116]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.134]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:bba871d020b1250b epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:bba871d020b1250b"}
12:12:27 [12:12:27.135]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba99c6f68f74eef6 epochNumber=0
{"provingJobId":"1:BASE_PARITY:ba99c6f68f74eef6"}
12:12:27 [12:12:27.136]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:bba871d020b1250b epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:bba871d020b1250b"}
12:12:27 [12:12:27.136]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa1d8649a6db9093 epochNumber=0
{"provingJobId":"1:BASE_PARITY:aa1d8649a6db9093"}
12:12:27 [12:12:27.153]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.158]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.159]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bb6cce63ad69938b epochNumber=2
{"provingJobId":"1:BASE_PARITY:bb6cce63ad69938b"}
12:12:27 [12:12:27.162]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwNpDUx/2 with map size 1073741824
12:12:27 [12:12:27.164]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.182]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:47185ebc3ce22846 epochNumber=1
{"provingJobId":"1:BASE_PARITY:47185ebc3ce22846"}
12:12:27 [12:12:27.184]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwNpDUx/1 with map size 1073741824
12:12:27 [12:12:27.186]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.207]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.208]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68e1f60b02352630 epochNumber=1
{"provingJobId":"1:BASE_PARITY:68e1f60b02352630"}
12:12:27 [12:12:27.209]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCaSD8g/1 with map size 1073741824
12:12:27 [12:12:27.214]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.235]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d78de3047ad7081 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d78de3047ad7081"}
12:12:27 [12:12:27.236]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test27DdiE/1 with map size 1073741824
12:12:27 [12:12:27.240]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:339b5588f64dd453 epochNumber=1
{"provingJobId":"1:BASE_PARITY:339b5588f64dd453"}
12:12:27 [12:12:27.275]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e76676e5f7074535 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e76676e5f7074535"}
12:12:27 [12:12:27.276]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test27DdiE/2 with map size 1073741824
12:12:27 [12:12:27.278]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.297]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e214f7dbbd1bc34c epochNumber=1
{"provingJobId":"1:BASE_PARITY:e214f7dbbd1bc34c"}
12:12:27 [12:12:27.309]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.310]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91256d96ee3c1e66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:91256d96ee3c1e66"}
12:12:27 [12:12:27.311]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testThVR4L/1 with map size 1073741824
12:12:27 [12:12:27.313]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.331]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:408e532f665df5c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:408e532f665df5c0"}
12:12:27 [12:12:27.339]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c37e749762017327 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c37e749762017327"}
12:12:27 [12:12:27.340]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testThVR4L/2 with map size 1073741824
12:12:27 [12:12:27.342]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.363]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d523eb77c51bc8e epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d523eb77c51bc8e"}
12:12:27 [12:12:27.375]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.376]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1e12fc12d08f98e epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1e12fc12d08f98e"}
12:12:27 [12:12:27.376]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testh59a36/1 with map size 1073741824
12:12:27 [12:12:27.380]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.396]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c6c5717adf466fbd epochNumber=1
{"provingJobId":"1:BASE_PARITY:c6c5717adf466fbd"}
12:12:27 [12:12:27.401]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68dcce7389f47983 epochNumber=2
{"provingJobId":"1:BASE_PARITY:68dcce7389f47983"}
12:12:27 [12:12:27.402]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testh59a36/2 with map size 1073741824
12:12:27 [12:12:27.406]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.423]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e50fa34b54b5051 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0e50fa34b54b5051"}
12:12:27 [12:12:27.432]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.433]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6d230df7c2ba9ec epochNumber=1
{"provingJobId":"1:BASE_PARITY:b6d230df7c2ba9ec"}
12:12:27 [12:12:27.435]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test56kcyM/1 with map size 1073741824
12:12:27 [12:12:27.438]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.462]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9815d77b0bcf4765 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9815d77b0bcf4765"}
12:12:27 [12:12:27.468]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b6d230df7c2ba9ec type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b6d230df7c2ba9ec"}
12:12:27 [12:12:27.482]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.483]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3aea0ed99b7bf8c9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3aea0ed99b7bf8c9"}
12:12:27 [12:12:27.483]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCw5s16/1 with map size 1073741824
12:12:27 [12:12:27.485]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.502]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f741d6e821d10cf6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f741d6e821d10cf6"}
12:12:27 [12:12:27.508]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3aea0ed99b7bf8c9 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:3aea0ed99b7bf8c9"}
12:12:27 [12:12:27.519]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe76eccb77832b0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe76eccb77832b0f"}
12:12:27 [12:12:27.521]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test58FStp/1 with map size 1073741824
12:12:27 [12:12:27.523]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.541]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a74dc9fcfd17da04 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a74dc9fcfd17da04"}
12:12:27 [12:12:27.547]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fe76eccb77832b0f type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:fe76eccb77832b0f"}
12:12:27 [12:12:27.551]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.552]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f9e28e25c3cb7391 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f9e28e25c3cb7391"}
12:12:27 [12:12:27.553]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8iFHyU/1 with map size 1073741824
12:12:27 [12:12:27.559]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.581]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f9e28e25c3cb7391
{"provingJobId":"1:BASE_PARITY:f9e28e25c3cb7391"}
12:12:27 [12:12:27.582]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f9e28e25c3cb7391 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f9e28e25c3cb7391"}
12:12:27 [12:12:27.588]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ceb3835c44ed1021 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ceb3835c44ed1021"}
12:12:27 [12:12:27.594]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f9e28e25c3cb7391 has already been completed
{"provingJobId":"1:BASE_PARITY:f9e28e25c3cb7391"}
12:12:27 [12:12:27.598]
INFO:
prover-client:proving-broker Proving Broker started
12:12:27 [12:12:27.599]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b22920d09124c6d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b22920d09124c6d3"}
12:12:27 [12:12:27.600]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqQePb9/1 with map size 1073741824
12:12:27 [12:12:27.602]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:27 [12:12:27.624]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd3c0cad93fee272 epochNumber=2
{"provingJobId":"1:BASE_PARITY:dd3c0cad93fee272"}
12:12:27 [12:12:27.625]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqQePb9/2 with map size 1073741824
12:12:27 [12:12:27.628]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:28 [12:12:28.699]
INFO:
prover-client:proving-broker Proving Broker started
12:12:28 [12:12:28.706]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b22920d09124c6d3 settled=false
{"provingJobId":"1:BASE_PARITY:b22920d09124c6d3","status":"pending"}
12:12:28 [12:12:28.708]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dd3c0cad93fee272 settled=false
{"provingJobId":"1:BASE_PARITY:dd3c0cad93fee272","status":"pending"}
12:12:28 [12:12:28.710]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b22920d09124c6d3 type=BASE_PARITY startedAt=1750767147648 older agent has taken job
{"provingJobId":"1:BASE_PARITY:b22920d09124c6d3"}
12:12:28 [12:12:28.710]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b22920d09124c6d3 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:b22920d09124c6d3"}
12:12:28 [12:12:28.715]
INFO:
prover-client:proving-broker Proving Broker started
12:12:28 [12:12:28.716]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b1548a2e08e0d1a epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b1548a2e08e0d1a"}
12:12:28 [12:12:28.717]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCvX11r/1 with map size 1073741824
12:12:28 [12:12:28.720]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:28 [12:12:28.740]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0c457850be4c083 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b0c457850be4c083"}
12:12:28 [12:12:28.742]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCvX11r/2 with map size 1073741824
12:12:28 [12:12:28.745]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:29 [12:12:29.781]
INFO:
prover-client:proving-broker Proving Broker started
12:12:29 [12:12:29.783]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2b1548a2e08e0d1a settled=false
{"provingJobId":"1:BASE_PARITY:2b1548a2e08e0d1a","status":"pending"}
12:12:29 [12:12:29.784]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b0c457850be4c083 settled=false
{"provingJobId":"1:BASE_PARITY:b0c457850be4c083","status":"pending"}
12:12:29 [12:12:29.785]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2b1548a2e08e0d1a type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:2b1548a2e08e0d1a"}
12:12:29 [12:12:29.790]
INFO:
prover-client:proving-broker Proving Broker started
12:12:29 [12:12:29.791]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b3de31dd8a7151c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b3de31dd8a7151c"}
12:12:29 [12:12:29.792]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0iIaMB/1 with map size 1073741824
12:12:29 [12:12:29.794]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:29 [12:12:29.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72ac9e9ca961b92e epochNumber=2
{"provingJobId":"1:BASE_PARITY:72ac9e9ca961b92e"}
12:12:29 [12:12:29.814]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0iIaMB/2 with map size 1073741824
12:12:29 [12:12:29.816]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:30 [12:12:30.835]
INFO:
prover-client:proving-broker Proving Broker started
12:12:30 [12:12:30.838]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1b3de31dd8a7151c settled=false
{"provingJobId":"1:BASE_PARITY:1b3de31dd8a7151c","status":"pending"}
12:12:30 [12:12:30.840]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:72ac9e9ca961b92e settled=false
{"provingJobId":"1:BASE_PARITY:72ac9e9ca961b92e","status":"pending"}
12:12:30 [12:12:30.842]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1b3de31dd8a7151c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1b3de31dd8a7151c"}
12:12:30 [12:12:30.842]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1b3de31dd8a7151c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:1b3de31dd8a7151c"}
12:12:30 [12:12:30.859]
INFO:
prover-client:proving-broker Proving Broker started
12:12:30 [12:12:30.860]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:087a1afe1a66b2f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:087a1afe1a66b2f2"}
12:12:30 [12:12:30.864]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5dXza6/1 with map size 1073741824
12:12:30 [12:12:30.867]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:30 [12:12:30.888]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d9f4b348f7db15fd epochNumber=2
{"provingJobId":"1:BASE_PARITY:d9f4b348f7db15fd"}
12:12:30 [12:12:30.891]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5dXza6/2 with map size 1073741824
12:12:30 [12:12:30.894]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:30 [12:12:30.919]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:087a1afe1a66b2f2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:087a1afe1a66b2f2"}
12:12:30 [12:12:30.928]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d9f4b348f7db15fd type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d9f4b348f7db15fd"}
12:12:30 [12:12:30.939]
INFO:
prover-client:proving-broker Proving Broker started
12:12:30 [12:12:30.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76095fd67ea11fb5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:76095fd67ea11fb5"}
12:12:30 [12:12:30.941]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyLLBqU/1 with map size 1073741824
12:12:30 [12:12:30.946]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:30 [12:12:30.971]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6d3d040ee145548 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b6d3d040ee145548"}
12:12:30 [12:12:30.978]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyLLBqU/2 with map size 1073741824
12:12:30 [12:12:30.982]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.001]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:76095fd67ea11fb5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:76095fd67ea11fb5"}
12:12:31 [12:12:31.001]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:76095fd67ea11fb5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:76095fd67ea11fb5"}
12:12:31 [12:12:31.011]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b6d3d040ee145548 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b6d3d040ee145548"}
12:12:31 [12:12:31.023]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.024]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7063be526b26bdd5 err=test error
{"provingJoId":"1:BASE_PARITY:7063be526b26bdd5"}
12:12:31 [12:12:31.028]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.030]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8a1820997ea25ccf not found
{"provingJobId":"1:BASE_PARITY:8a1820997ea25ccf"}
12:12:31 [12:12:31.036]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.037]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a663c146e0ebc025 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a663c146e0ebc025"}
12:12:31 [12:12:31.037]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testP3R3Pt/1 with map size 1073741824
12:12:31 [12:12:31.039]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.069]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.071]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef8b47e6d0be1de7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef8b47e6d0be1de7"}
12:12:31 [12:12:31.073]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcQJ7YJ/1 with map size 1073741824
12:12:31 [12:12:31.078]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.224]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ef8b47e6d0be1de7 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:ef8b47e6d0be1de7"}
12:12:31 [12:12:31.236]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:209c9b2879f58617 epochNumber=1
{"provingJobId":"1:BASE_PARITY:209c9b2879f58617"}
12:12:31 [12:12:31.238]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHw6USh/1 with map size 1073741824
12:12:31 [12:12:31.241]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.390]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:209c9b2879f58617 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:209c9b2879f58617"}
12:12:31 [12:12:31.392]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b9d39b0b14ee298 epochNumber=10
{"provingJobId":"1:BASE_PARITY:1b9d39b0b14ee298"}
12:12:31 [12:12:31.393]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testHw6USh/10 with map size 1073741824
12:12:31 [12:12:31.395]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.415]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:31 [12:12:31.416]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:12:31 [12:12:31.544]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.545]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:147bd48a7d57a594 epochNumber=1
{"provingJobId":"1:BASE_PARITY:147bd48a7d57a594"}
12:12:31 [12:12:31.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrBpkuy/1 with map size 1073741824
12:12:31 [12:12:31.547]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.722]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:147bd48a7d57a594 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:147bd48a7d57a594"}
12:12:31 [12:12:31.772]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.773]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:021b81074e42d0bd epochNumber=1
{"provingJobId":"1:BASE_PARITY:021b81074e42d0bd"}
12:12:31 [12:12:31.774]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaKXRDP/1 with map size 1073741824
12:12:31 [12:12:31.779]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.803]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:021b81074e42d0bd type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:021b81074e42d0bd"}
12:12:31 [12:12:31.808]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.809]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f68f0da0192cc62c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f68f0da0192cc62c"}
12:12:31 [12:12:31.810]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZcvDOx/1 with map size 1073741824
12:12:31 [12:12:31.812]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.829]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f68f0da0192cc62c type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f68f0da0192cc62c"}
12:12:31 [12:12:31.830]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f68f0da0192cc62c type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:f68f0da0192cc62c"}
12:12:31 [12:12:31.839]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.840]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d95593bf42996d8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d95593bf42996d8"}
12:12:31 [12:12:31.840]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPFQjN6/1 with map size 1073741824
12:12:31 [12:12:31.844]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.873]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6d95593bf42996d8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6d95593bf42996d8"}
12:12:31 [12:12:31.886]
INFO:
prover-client:proving-broker Proving Broker started
12:12:31 [12:12:31.887]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:093738b3503dc09a epochNumber=1
{"provingJobId":"1:BASE_PARITY:093738b3503dc09a"}
12:12:31 [12:12:31.889]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrGDKzc/1 with map size 1073741824
12:12:31 [12:12:31.893]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.915]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:093738b3503dc09a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:093738b3503dc09a"}
12:12:31 [12:12:31.916]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f2bf168de4ed281 epochNumber=3
{"provingJobId":"1:BASE_PARITY:6f2bf168de4ed281"}
12:12:31 [12:12:31.918]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testrGDKzc/3 with map size 1073741824
12:12:31 [12:12:31.922]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:31 [12:12:31.965]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:31 [12:12:31.966]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:12:31 [12:12:31.973]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:093738b3503dc09a err=test error
{"provingJoId":"1:BASE_PARITY:093738b3503dc09a"}
12:12:31 [12:12:31.982]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyUZWPx/1 with map size 1073741824
12:12:31 [12:12:31.986]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.008]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyUZWPx/2 with map size 1073741824
12:12:32 [12:12:32.011]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.026]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.029]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6edbe2be58a11812 settled=false
{"provingJobId":"1:BASE_PARITY:6edbe2be58a11812","status":"pending"}
12:12:32 [12:12:32.031]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8417f3e953f3330b settled=false
{"provingJobId":"1:BASE_PARITY:8417f3e953f3330b","status":"pending"}
12:12:32 [12:12:32.038]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testemNb3y/1 with map size 1073741824
12:12:32 [12:12:32.043]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.066]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testemNb3y/2 with map size 1073741824
12:12:32 [12:12:32.071]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.105]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.108]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5ea7f2719c04b776 settled=true
{"provingJobId":"1:BASE_PARITY:5ea7f2719c04b776","status":"fulfilled"}
12:12:32 [12:12:32.110]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:4d12f83b6ffa4d5b settled=true
{"provingJobId":"2:BASE_PARITY:4d12f83b6ffa4d5b","status":"fulfilled"}
12:12:32 [12:12:32.117]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCsvlnm/1 with map size 1073741824
12:12:32 [12:12:32.121]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.151]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCsvlnm/2 with map size 1073741824
12:12:32 [12:12:32.159]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.181]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.183]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7bb4504bdaf28b3e settled=true
{"provingJobId":"1:BASE_PARITY:7bb4504bdaf28b3e","status":"fulfilled"}
12:12:32 [12:12:32.186]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4599ebfcbc78544f settled=false
{"provingJobId":"1:BASE_PARITY:4599ebfcbc78544f","status":"pending"}
12:12:32 [12:12:32.193]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.193]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee28e4ad282b0e4e epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee28e4ad282b0e4e"}
12:12:32 [12:12:32.196]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxbalOF/1 with map size 1073741824
12:12:32 [12:12:32.200]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.226]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.227]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f4a4251ea2ca736 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f4a4251ea2ca736"}
12:12:32 [12:12:32.228]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:8f4a4251ea2ca736: Error: db error: Error: db error
12:12:32 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:8f4a4251ea2ca736"}
12:12:32 [12:12:32.234]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.234]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:33be2793c4c24447 epochNumber=1
{"provingJobId":"1:BASE_PARITY:33be2793c4c24447"}
12:12:32 [12:12:32.236]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBONMLH/1 with map size 1073741824
12:12:32 [12:12:32.241]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.273]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:33be2793c4c24447 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:33be2793c4c24447"}
12:12:32 [12:12:32.274]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:33be2793c4c24447 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:33be2793c4c24447"}
12:12:32 [12:12:32.289]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.290]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:29d0147a4dbbe0f6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:29d0147a4dbbe0f6"}
12:12:32 [12:12:32.292]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZ8PszN/1 with map size 1073741824
12:12:32 [12:12:32.297]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.324]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:29d0147a4dbbe0f6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:29d0147a4dbbe0f6"}
12:12:32 [12:12:32.325]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:29d0147a4dbbe0f6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:29d0147a4dbbe0f6"}
12:12:32 [12:12:32.326]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:29d0147a4dbbe0f6: Error: db error
12:12:32 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:29d0147a4dbbe0f6"}
12:12:32 [12:12:32.331]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.332]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b298fb05f52fdecb epochNumber=1
{"provingJobId":"1:BASE_PARITY:b298fb05f52fdecb"}
12:12:32 [12:12:32.333]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCwIscw/1 with map size 1073741824
12:12:32 [12:12:32.336]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.374]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b298fb05f52fdecb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b298fb05f52fdecb"}
12:12:32 [12:12:32.375]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b298fb05f52fdecb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b298fb05f52fdecb"}
12:12:32 [12:12:32.390]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.391]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd9b4ce8ea8eacf8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd9b4ce8ea8eacf8"}
12:12:32 [12:12:32.394]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVZvzwO/1 with map size 1073741824
12:12:32 [12:12:32.402]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.422]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd9b4ce8ea8eacf8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fd9b4ce8ea8eacf8"}
12:12:32 [12:12:32.422]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fd9b4ce8ea8eacf8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fd9b4ce8ea8eacf8"}
12:12:32 [12:12:32.423]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:fd9b4ce8ea8eacf8 jobErr=test error: Error: db error
12:12:32 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:fd9b4ce8ea8eacf8"}
12:12:32 [12:12:32.429]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.429]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:58788fa6ee7611ff not found
{"provingJobId":"1:BASE_PARITY:58788fa6ee7611ff"}
12:12:32 [12:12:32.435]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.436]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d78209eece14f42e err=test error
{"provingJoId":"1:BASE_PARITY:d78209eece14f42e"}
12:12:32 [12:12:32.441]
INFO:
prover-client:proving-broker Proving Broker started
12:12:32 [12:12:32.451]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3504112f0cc8a53 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3504112f0cc8a53"}
12:12:32 [12:12:32.454]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2ray43/1 with map size 1073741824
12:12:32 [12:12:32.459]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.480]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a3504112f0cc8a53 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a3504112f0cc8a53"}
12:12:32 [12:12:32.481]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a3504112f0cc8a53 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a3504112f0cc8a53"}
12:12:32 [12:12:32.497]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:3aa1423597c66f71 epochNumber=2
{"provingJobId":"2:BASE_PARITY:3aa1423597c66f71"}
12:12:32 [12:12:32.498]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2ray43/2 with map size 1073741824
12:12:32 [12:12:32.500]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.519]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:3aa1423597c66f71 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:3aa1423597c66f71"}
12:12:32 [12:12:32.519]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:3aa1423597c66f71 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:3aa1423597c66f71"}
12:12:32 [12:12:32.586]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:88237f9623133e59 epochNumber=3
{"provingJobId":"3:BASE_PARITY:88237f9623133e59"}
12:12:32 [12:12:32.587]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test2ray43/3 with map size 1073741824
12:12:32 [12:12:32.591]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.620]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:32 [12:12:32.621]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:12:32 [12:12:32.679]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:9b56d58fa1ce7f3f epochNumber=4
{"provingJobId":"4:BASE_PARITY:9b56d58fa1ce7f3f"}
12:12:32 [12:12:32.680]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-test2ray43/4 with map size 1073741824
12:12:32 [12:12:32.682]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.725]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:32 [12:12:32.726]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
12:12:32 [12:12:32.776]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:ecc1631735f3527f epochNumber=5
{"provingJobId":"5:BASE_PARITY:ecc1631735f3527f"}
12:12:32 [12:12:32.778]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-test2ray43/5 with map size 1073741824
12:12:32 [12:12:32.786]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:12:32 [12:12:32.829]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:12:32 [12:12:32.830]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
12:12:32
PASS src/proving_broker/proving_broker.test.ts (
13.922 s)
12:12:32 ProvingBroker
12:12:32 Producer API
12:12:32
✓ refuses stale jobs (47 ms)
12:12:32
✓ enqueues jobs (6 ms)
12:12:32
✓ ignores duplicate jobs (4 ms)
12:12:32
✓ reports correct status when enqueuing repeat jobs (4 ms)
12:12:32
✓ throws an error in case of duplicate job IDs (26 ms)
12:12:32
✓ returns not-found status for non-existing jobs (2 ms)
12:12:32
✓ cancels jobs in queue (3 ms)
12:12:32
✓ cancels jobs in-progress (2 ms)
12:12:32
✓ returns job result if successful (3 ms)
12:12:32
✓ returns job error if failed (2 ms)
12:12:32
✓ correctly returns job status for concurrent writes (12 ms)
12:12:32
✓ refuses stale jobs (56 ms)
12:12:32
✓ enqueues jobs (38 ms)
12:12:32
✓ ignores duplicate jobs (29 ms)
12:12:32
✓ reports correct status when enqueuing repeat jobs (44 ms)
12:12:32
✓ throws an error in case of duplicate job IDs (29 ms)
12:12:32
✓ returns not-found status for non-existing jobs (3 ms)
12:12:32
✓ cancels jobs in queue (46 ms)
12:12:32
✓ cancels jobs in-progress (42 ms)
12:12:32
✓ returns job result if successful (42 ms)
12:12:32
✓ returns job error if failed (54 ms)
12:12:32
✓ correctly returns job status for concurrent writes (45 ms)
12:12:32 Consumer API
12:12:32
✓ returns undefined if no jobs are available (2 ms)
12:12:32
✓ returns jobs in priority order (4 ms)
12:12:32
✓ returns undefined if no jobs are available for the given allowList (2 ms)
12:12:32
✓ returns a job if it is in the allowList (3 ms)
12:12:32
✓ returns the most important job if it is in the allowList (3 ms)
12:12:32
✓ returns any job if filter is empty (3 ms)
12:12:32
✓ returns a new job when reporting job success (3 ms)
12:12:32
✓ returns a new job when reporting permanent error (2 ms)
12:12:32
✓ returns a new job when reporting retry-able error (2 ms)
12:12:32
✓ returns a new job when reporting progress if current one is cancelled (4 ms)
12:12:32
✓ returns a new job if job is already in progress elsewhere (1060 ms)
12:12:32
✓ avoids sending the same job to a new agent after a restart (1005 ms)
12:12:32
✓ avoids sending a completed job to a new agent after a restart (1006 ms)
12:12:32
✓ tracks job result if in progress (4 ms)
12:12:32
✓ tracks job result even if job is in queue (3 ms)
12:12:32
✓ ignores reported job error if unknown job (1 ms)
12:12:32
✓ ignores job result if unknown job (2 ms)
12:12:32
✓ returns undefined if no jobs are available (3 ms)
12:12:32
✓ returns jobs in priority order (50 ms)
12:12:32
✓ returns undefined if no jobs are available for the given allowList (29 ms)
12:12:32
✓ returns a job if it is in the allowList (72 ms)
12:12:32
✓ returns the most important job if it is in the allowList (67 ms)
12:12:32
✓ returns any job if filter is empty (58 ms)
12:12:32
✓ returns a new job when reporting job success (45 ms)
12:12:32
✓ returns a new job when reporting permanent error (40 ms)
12:12:32
✓ returns a new job when reporting retry-able error (32 ms)
12:12:32
✓ returns a new job when reporting progress if current one is cancelled (47 ms)
12:12:32
✓ returns a new job if job is already in progress elsewhere (1117 ms)
12:12:32
✓ avoids sending the same job to a new agent after a restart (1075 ms)
12:12:32
✓ avoids sending a completed job to a new agent after a restart (1065 ms)
12:12:32
✓ tracks job result if in progress (81 ms)
12:12:32
✓ tracks job result even if job is in queue (85 ms)
12:12:32
✓ ignores reported job error if unknown job (5 ms)
12:12:32
✓ ignores job result if unknown job (5 ms)
12:12:32 Timeouts
12:12:32
✓ tracks in progress jobs (2 ms)
12:12:32
✓ re-enqueues jobs that time out (110 ms)
12:12:32
✓ cancel stale jobs that time out (239 ms)
12:12:32
✓ keeps the jobs in progress while it is alive (204 ms)
12:12:32
✓ tracks in progress jobs (32 ms)
12:12:32
✓ re-enqueues jobs that time out (166 ms)
12:12:32
✓ cancel stale jobs that time out (308 ms)
12:12:32
✓ keeps the jobs in progress while it is alive (230 ms)
12:12:32 Retries
12:12:32
✓ retries jobs (3 ms)
12:12:32
✓ retries up to a maximum number of times (2 ms)
12:12:32
✓ passing retry=false does not retry (1 ms)
12:12:32
✓ does not retry if job is stale (29 ms)
12:12:32
✓ retries jobs (35 ms)
12:12:32
✓ retries up to a maximum number of times (31 ms)
12:12:32
✓ passing retry=false does not retry (46 ms)
12:12:32
✓ does not retry if job is stale (92 ms)
12:12:32 Database management
12:12:32
✓ re-enqueues proof requests on start (7 ms)
12:12:32
✓ restores proof results on start (3 ms)
12:12:32
✓ only re-enqueues unfinished jobs (3 ms)
12:12:32
✓ saves job when enqueued (4 ms)
12:12:32
✓ does not retain job if database fails to save (3 ms)
12:12:32
✓ saves job result (3 ms)
12:12:32
✓ saves result even if database fails to save (3 ms)
12:12:32
✓ saves job error (3 ms)
12:12:32
✓ saves job error even if database fails to save (3 ms)
12:12:32
✓ does not save job result if job is unknown (2 ms)
12:12:32
✓ does not save job error if job is unknown (1 ms)
12:12:32
✓ cleans up old jobs periodically (275 ms)
12:12:32
✓ re-enqueues proof requests on start (59 ms)
12:12:32
✓ restores proof results on start (78 ms)
12:12:32
✓ only re-enqueues unfinished jobs (75 ms)
12:12:32
✓ saves job when enqueued (32 ms)
12:12:32
✓ does not retain job if database fails to save (8 ms)
12:12:32
✓ saves job result (52 ms)
12:12:32
✓ saves result even if database fails to save (44 ms)
12:12:32
✓ saves job error (57 ms)
12:12:32
✓ saves job error even if database fails to save (39 ms)
12:12:32
✓ does not save job result if job is unknown (5 ms)
12:12:32
✓ does not save job error if job is unknown (6 ms)
12:12:32
✓ cleans up old jobs periodically (431 ms)
12:12:32
12:12:32
Test Suites: 1 passed, 1 total
12:12:32
Tests: 96 passed, 96 total
12:12:32
Snapshots: 0 total
12:12:32
Time: 14.029 s
12:12:32
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
12:12:32
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?