Parent Log:
http://ci.aztec-labs.com/64815db951c4de2c
Command: 6a9c991c4402d14c yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/47c749527b1c3cce3edd9defedc46e89ea00c69e
Env: REF_NAME=gh-readonly-queue/next/pr-15009-00cd67e98e76d3202a0219baf301d59ef4211872 CURRENT_VERSION=0.87.6 CI_FULL=0
Date: Fri Jun 13 16:00:20 UTC 2025
System: ARCH=arm64 CPUS=64 MEM=247Gi HOSTNAME=pr-15009_arm64_a1-fast
Resources: CPU_LIST=0-63 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
16:00:22 [16:00:22.955]
INFO:
prover-client:proving-broker Proving Broker started
16:00:22 [16:00:22.961]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ddd94f797c3408b5 epochNumber=42
{"provingJobId":"1:BASE_PARITY:ddd94f797c3408b5"}
16:00:22 [16:00:22.963]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:0454f14aee69513d epochNumber=1
{"provingJobId":"1:BASE_PARITY:0454f14aee69513d"}
16:00:22 [16:00:22.987]
INFO:
prover-client:proving-broker Proving Broker started
16:00:22 [16:00:22.988]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce70febc050ce25d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ce70febc050ce25d"}
16:00:22 [16:00:22.989]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:552397bd9a899f5d epochNumber=1
{"provingJobId":"1:BASE_PARITY:552397bd9a899f5d"}
16:00:22 [16:00:22.990]
INFO:
prover-client:proving-broker Proving Broker started
16:00:22 [16:00:22.991]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a2feec7449370de epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a2feec7449370de"}
16:00:22 [16:00:22.992]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0a2feec7449370de epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0a2feec7449370de"}
16:00:22 [16:00:22.994]
INFO:
prover-client:proving-broker Proving Broker started
16:00:22 [16:00:22.995]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee386164e0d7d473 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee386164e0d7d473"}
16:00:22 [16:00:22.996]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ee386164e0d7d473 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ee386164e0d7d473"}
16:00:22 [16:00:22.997]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ee386164e0d7d473 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ee386164e0d7d473"}
16:00:22 [16:00:22.997]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ee386164e0d7d473 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ee386164e0d7d473"}
16:00:22 [16:00:22.998]
INFO:
prover-client:proving-broker Proving Broker started
16:00:22 [16:00:22.999]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5222ddd0624972a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5222ddd0624972a5"}
16:00:23 [16:00:23.021]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.022]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.022]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:25b8d5c6e263e69d epochNumber=1
{"provingJobId":"1:BASE_PARITY:25b8d5c6e263e69d"}
16:00:23 [16:00:23.023]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:25b8d5c6e263e69d
{"provingJobId":"1:BASE_PARITY:25b8d5c6e263e69d"}
16:00:23 [16:00:23.023]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:25b8d5c6e263e69d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:25b8d5c6e263e69d"}
16:00:23 [16:00:23.024]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:25b8d5c6e263e69d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:25b8d5c6e263e69d"}
16:00:23 [16:00:23.025]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70ab208ae3a681ee epochNumber=1
{"provingJobId":"1:BASE_PARITY:70ab208ae3a681ee"}
16:00:23 [16:00:23.026]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:70ab208ae3a681ee
{"provingJobId":"1:BASE_PARITY:70ab208ae3a681ee"}
16:00:23 [16:00:23.026]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:70ab208ae3a681ee type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:70ab208ae3a681ee"}
16:00:23 [16:00:23.027]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.028]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4a6aa3b7f0c26091 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4a6aa3b7f0c26091"}
16:00:23 [16:00:23.029]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4a6aa3b7f0c26091 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4a6aa3b7f0c26091"}
16:00:23 [16:00:23.029]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4a6aa3b7f0c26091 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4a6aa3b7f0c26091"}
16:00:23 [16:00:23.030]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.031]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fadc3d2af935fe33 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fadc3d2af935fe33"}
16:00:23 [16:00:23.031]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fadc3d2af935fe33 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fadc3d2af935fe33"}
16:00:23 [16:00:23.031]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fadc3d2af935fe33 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:fadc3d2af935fe33"}
16:00:23 [16:00:23.032]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba37bdee39eee15d epochNumber=0
{"provingJobId":"1:BASE_PARITY:ba37bdee39eee15d"}
16:00:23 [16:00:23.033]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ba37bdee39eee15d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ba37bdee39eee15d"}
16:00:23 [16:00:23.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ada1ab96e31a07fd epochNumber=0
{"provingJobId":"1:BASE_PARITY:ada1ab96e31a07fd"}
16:00:23 [16:00:23.033]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ba37bdee39eee15d epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:ba37bdee39eee15d"}
16:00:23 [16:00:23.034]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a5fbea629d49c83 epochNumber=0
{"provingJobId":"1:BASE_PARITY:5a5fbea629d49c83"}
16:00:23 [16:00:23.035]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.036]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d0c79c92384ff725 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d0c79c92384ff725"}
16:00:23 [16:00:23.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6eeac526c60d868a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6eeac526c60d868a"}
16:00:23 [16:00:23.037]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.038]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55e3853870b80725 epochNumber=1
{"provingJobId":"1:BASE_PARITY:55e3853870b80725"}
16:00:23 [16:00:23.039]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.039]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:415932395c8db239 epochNumber=1
{"provingJobId":"1:BASE_PARITY:415932395c8db239"}
16:00:23 [16:00:23.039]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2dd8de15ea4a4be2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2dd8de15ea4a4be2"}
16:00:23 [16:00:23.039]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c370980bd11ae97b epochNumber=2
{"provingJobId":"1:BASE_PARITY:c370980bd11ae97b"}
16:00:23 [16:00:23.040]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77b0ff67aa040a29 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77b0ff67aa040a29"}
16:00:23 [16:00:23.041]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.041]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b576f608e69415a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b576f608e69415a5"}
16:00:23 [16:00:23.041]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb00540ac835fd5f epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb00540ac835fd5f"}
16:00:23 [16:00:23.042]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6813cf796a0cac3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d6813cf796a0cac3"}
16:00:23 [16:00:23.042]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c988f6bc5906aa93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c988f6bc5906aa93"}
16:00:23 [16:00:23.043]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.043]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1faacf7a6633677e epochNumber=1
{"provingJobId":"1:BASE_PARITY:1faacf7a6633677e"}
16:00:23 [16:00:23.043]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:262d70c9bad0197c epochNumber=1
{"provingJobId":"1:BASE_PARITY:262d70c9bad0197c"}
16:00:23 [16:00:23.044]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a7e7dd3373dab76 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9a7e7dd3373dab76"}
16:00:23 [16:00:23.044]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9401db9bd1f39cc9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9401db9bd1f39cc9"}
16:00:23 [16:00:23.045]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.045]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b7e55927ce7adff8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b7e55927ce7adff8"}
16:00:23 [16:00:23.046]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c22e32bb0a9acc75 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c22e32bb0a9acc75"}
16:00:23 [16:00:23.046]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b7e55927ce7adff8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b7e55927ce7adff8"}
16:00:23 [16:00:23.047]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.047]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca6dbbcbf00673e0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca6dbbcbf00673e0"}
16:00:23 [16:00:23.048]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:957fe55864c84fec epochNumber=1
{"provingJobId":"1:BASE_PARITY:957fe55864c84fec"}
16:00:23 [16:00:23.048]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ca6dbbcbf00673e0 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:ca6dbbcbf00673e0"}
16:00:23 [16:00:23.049]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.049]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31072d1b6021b0e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:31072d1b6021b0e1"}
16:00:23 [16:00:23.050]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a39305ef6a82f7e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a39305ef6a82f7e3"}
16:00:23 [16:00:23.050]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:31072d1b6021b0e1 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:31072d1b6021b0e1"}
16:00:23 [16:00:23.051]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.051]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0febe66300e95fc1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0febe66300e95fc1"}
16:00:23 [16:00:23.052]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0febe66300e95fc1
{"provingJobId":"1:BASE_PARITY:0febe66300e95fc1"}
16:00:23 [16:00:23.052]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0febe66300e95fc1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:0febe66300e95fc1"}
16:00:23 [16:00:23.052]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5265d1784bb5459d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5265d1784bb5459d"}
16:00:23 [16:00:23.053]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0febe66300e95fc1 has already been completed
{"provingJobId":"1:BASE_PARITY:0febe66300e95fc1"}
16:00:23 [16:00:23.054]
INFO:
prover-client:proving-broker Proving Broker started
16:00:23 [16:00:23.054]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:db506ad386eb44d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:db506ad386eb44d1"}
16:00:23 [16:00:23.054]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82a731be01976e46 epochNumber=2
{"provingJobId":"1:BASE_PARITY:82a731be01976e46"}
16:00:24 [16:00:24.105]
INFO:
prover-client:proving-broker Proving Broker started
16:00:24 [16:00:24.106]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:db506ad386eb44d1 settled=false
{"provingJobId":"1:BASE_PARITY:db506ad386eb44d1","status":"pending"}
16:00:24 [16:00:24.106]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:82a731be01976e46 settled=false
{"provingJobId":"1:BASE_PARITY:82a731be01976e46","status":"pending"}
16:00:24 [16:00:24.107]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:db506ad386eb44d1 type=BASE_PARITY startedAt=1749830423055 older agent has taken job
{"provingJobId":"1:BASE_PARITY:db506ad386eb44d1"}
16:00:24 [16:00:24.107]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:db506ad386eb44d1 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:db506ad386eb44d1"}
16:00:24 [16:00:24.108]
INFO:
prover-client:proving-broker Proving Broker started
16:00:24 [16:00:24.108]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b88a9380da78ec0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b88a9380da78ec0"}
16:00:24 [16:00:24.108]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba9a63babb275dfe epochNumber=2
{"provingJobId":"1:BASE_PARITY:ba9a63babb275dfe"}
16:00:25 [16:00:25.109]
INFO:
prover-client:proving-broker Proving Broker started
16:00:25 [16:00:25.109]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1b88a9380da78ec0 settled=false
{"provingJobId":"1:BASE_PARITY:1b88a9380da78ec0","status":"pending"}
16:00:25 [16:00:25.110]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ba9a63babb275dfe settled=false
{"provingJobId":"1:BASE_PARITY:ba9a63babb275dfe","status":"pending"}
16:00:25 [16:00:25.110]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1b88a9380da78ec0 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:1b88a9380da78ec0"}
16:00:25 [16:00:25.111]
INFO:
prover-client:proving-broker Proving Broker started
16:00:25 [16:00:25.112]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a93188fea6299bd6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a93188fea6299bd6"}
16:00:25 [16:00:25.112]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb3eca5281dad285 epochNumber=2
{"provingJobId":"1:BASE_PARITY:cb3eca5281dad285"}
16:00:26 [16:00:26.113]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.113]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a93188fea6299bd6 settled=false
{"provingJobId":"1:BASE_PARITY:a93188fea6299bd6","status":"pending"}
16:00:26 [16:00:26.114]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cb3eca5281dad285 settled=false
{"provingJobId":"1:BASE_PARITY:cb3eca5281dad285","status":"pending"}
16:00:26 [16:00:26.114]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a93188fea6299bd6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a93188fea6299bd6"}
16:00:26 [16:00:26.114]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a93188fea6299bd6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:a93188fea6299bd6"}
16:00:26 [16:00:26.115]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.116]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48aef14d48153de2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48aef14d48153de2"}
16:00:26 [16:00:26.116]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46da4517a5d994f8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:46da4517a5d994f8"}
16:00:26 [16:00:26.117]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:48aef14d48153de2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:48aef14d48153de2"}
16:00:26 [16:00:26.117]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:46da4517a5d994f8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:46da4517a5d994f8"}
16:00:26 [16:00:26.118]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:920140d844f3dc45 epochNumber=1
{"provingJobId":"1:BASE_PARITY:920140d844f3dc45"}
16:00:26 [16:00:26.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1cb94e47977be11e epochNumber=2
{"provingJobId":"1:BASE_PARITY:1cb94e47977be11e"}
16:00:26 [16:00:26.119]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:920140d844f3dc45 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:920140d844f3dc45"}
16:00:26 [16:00:26.119]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:920140d844f3dc45 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:920140d844f3dc45"}
16:00:26 [16:00:26.120]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1cb94e47977be11e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1cb94e47977be11e"}
16:00:26 [16:00:26.121]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.121]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:28b863ed2a42d440 err=test error
{"provingJoId":"1:BASE_PARITY:28b863ed2a42d440"}
16:00:26 [16:00:26.122]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.122]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2065e61060ceba32 not found
{"provingJobId":"1:BASE_PARITY:2065e61060ceba32"}
16:00:26 [16:00:26.123]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c1585b023371c16e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c1585b023371c16e"}
16:00:26 [16:00:26.125]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.125]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb4830851c7581cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb4830851c7581cb"}
16:00:26 [16:00:26.226]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eb4830851c7581cb timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:eb4830851c7581cb"}
16:00:26 [16:00:26.232]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.233]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3da4a1333858edc epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3da4a1333858edc"}
16:00:26 [16:00:26.333]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f3da4a1333858edc timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:f3da4a1333858edc"}
16:00:26 [16:00:26.340]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ffcc32ac627d3063 epochNumber=10
{"provingJobId":"1:BASE_PARITY:ffcc32ac627d3063"}
16:00:26 [16:00:26.359]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:26 [16:00:26.466]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.467]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a0358bc21cbdaea epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a0358bc21cbdaea"}
16:00:26 [16:00:26.641]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0a0358bc21cbdaea timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:0a0358bc21cbdaea"}
16:00:26 [16:00:26.669]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.670]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6808c6fcf66227c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6808c6fcf66227c6"}
16:00:26 [16:00:26.670]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:6808c6fcf66227c6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:6808c6fcf66227c6"}
16:00:26 [16:00:26.671]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.672]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7db93cfd06ac8708 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7db93cfd06ac8708"}
16:00:26 [16:00:26.672]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:7db93cfd06ac8708 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:7db93cfd06ac8708"}
16:00:26 [16:00:26.673]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7db93cfd06ac8708 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:7db93cfd06ac8708"}
16:00:26 [16:00:26.674]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.674]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd48ca3213eca98e epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd48ca3213eca98e"}
16:00:26 [16:00:26.674]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bd48ca3213eca98e type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bd48ca3213eca98e"}
16:00:26 [16:00:26.675]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.676]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8ba879fdb467797 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f8ba879fdb467797"}
16:00:26 [16:00:26.676]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f8ba879fdb467797 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f8ba879fdb467797"}
16:00:26 [16:00:26.677]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02219b3c3dc06ba8 epochNumber=3
{"provingJobId":"1:BASE_PARITY:02219b3c3dc06ba8"}
16:00:26 [16:00:26.701]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:26 [16:00:26.701]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f8ba879fdb467797 err=test error
{"provingJoId":"1:BASE_PARITY:f8ba879fdb467797"}
16:00:26 [16:00:26.702]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.703]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d693bda64517e97e settled=false
{"provingJobId":"1:BASE_PARITY:d693bda64517e97e","status":"pending"}
16:00:26 [16:00:26.703]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:27734a7cb2b505a9 settled=false
{"provingJobId":"1:BASE_PARITY:27734a7cb2b505a9","status":"pending"}
16:00:26 [16:00:26.705]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.705]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:869ba4dbfebbca5a settled=true
{"provingJobId":"1:BASE_PARITY:869ba4dbfebbca5a","status":"fulfilled"}
16:00:26 [16:00:26.705]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:ec8e78e2c78d02c6 settled=true
{"provingJobId":"2:BASE_PARITY:ec8e78e2c78d02c6","status":"fulfilled"}
16:00:26 [16:00:26.706]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.706]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1131ed86aca3a747 settled=true
{"provingJobId":"1:BASE_PARITY:1131ed86aca3a747","status":"fulfilled"}
16:00:26 [16:00:26.706]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a2c92b4f30e9809f settled=false
{"provingJobId":"1:BASE_PARITY:a2c92b4f30e9809f","status":"pending"}
16:00:26 [16:00:26.707]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.708]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5984ebfb53c7018d epochNumber=1
{"provingJobId":"1:BASE_PARITY:5984ebfb53c7018d"}
16:00:26 [16:00:26.710]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.711]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b8729f001652e55 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b8729f001652e55"}
16:00:26 [16:00:26.711]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:6b8729f001652e55: Error: db error: Error: db error
16:00:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:6b8729f001652e55"}
16:00:26 [16:00:26.712]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.713]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c8ee4d184dac7336 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c8ee4d184dac7336"}
16:00:26 [16:00:26.713]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c8ee4d184dac7336 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:c8ee4d184dac7336"}
16:00:26 [16:00:26.713]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c8ee4d184dac7336 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:c8ee4d184dac7336"}
16:00:26 [16:00:26.714]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.714]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55b4c1cd41f7969b epochNumber=1
{"provingJobId":"1:BASE_PARITY:55b4c1cd41f7969b"}
16:00:26 [16:00:26.715]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:55b4c1cd41f7969b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:55b4c1cd41f7969b"}
16:00:26 [16:00:26.715]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:55b4c1cd41f7969b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:55b4c1cd41f7969b"}
16:00:26 [16:00:26.715]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:55b4c1cd41f7969b: Error: db error
16:00:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:55b4c1cd41f7969b"}
16:00:26 [16:00:26.716]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.717]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0996490e9a55c0ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:0996490e9a55c0ad"}
16:00:26 [16:00:26.717]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0996490e9a55c0ad type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0996490e9a55c0ad"}
16:00:26 [16:00:26.717]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0996490e9a55c0ad type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0996490e9a55c0ad"}
16:00:26 [16:00:26.718]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.719]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ab944f883371241 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ab944f883371241"}
16:00:26 [16:00:26.719]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8ab944f883371241 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8ab944f883371241"}
16:00:26 [16:00:26.719]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8ab944f883371241 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8ab944f883371241"}
16:00:26 [16:00:26.719]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:8ab944f883371241 jobErr=test error: Error: db error
16:00:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:8ab944f883371241"}
16:00:26 [16:00:26.720]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.721]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5e3eea59042f5ebb not found
{"provingJobId":"1:BASE_PARITY:5e3eea59042f5ebb"}
16:00:26 [16:00:26.722]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.723]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e77998599da2a16e err=test error
{"provingJoId":"1:BASE_PARITY:e77998599da2a16e"}
16:00:26 [16:00:26.724]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.735]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24458df522ae3117 epochNumber=1
{"provingJobId":"1:BASE_PARITY:24458df522ae3117"}
16:00:26 [16:00:26.735]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:24458df522ae3117 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:24458df522ae3117"}
16:00:26 [16:00:26.735]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:24458df522ae3117 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:24458df522ae3117"}
16:00:26 [16:00:26.745]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:3bdf014ac8612c71 epochNumber=2
{"provingJobId":"2:BASE_PARITY:3bdf014ac8612c71"}
16:00:26 [16:00:26.745]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:3bdf014ac8612c71 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:3bdf014ac8612c71"}
16:00:26 [16:00:26.746]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:3bdf014ac8612c71 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:3bdf014ac8612c71"}
16:00:26 [16:00:26.806]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:ecf5818f277cb48a epochNumber=3
{"provingJobId":"3:BASE_PARITY:ecf5818f277cb48a"}
16:00:26 [16:00:26.825]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:26 [16:00:26.867]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:f59ff5b4b6ccee70 epochNumber=4
{"provingJobId":"4:BASE_PARITY:f59ff5b4b6ccee70"}
16:00:26 [16:00:26.876]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:26 [16:00:26.928]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:810d75dc5d21490c epochNumber=5
{"provingJobId":"5:BASE_PARITY:810d75dc5d21490c"}
16:00:26 [16:00:26.953]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:26 [16:00:26.981]
INFO:
prover-client:proving-broker Proving Broker started
16:00:26 [16:00:26.982]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60237a98699eedc4 epochNumber=42
{"provingJobId":"1:BASE_PARITY:60237a98699eedc4"}
16:00:26 [16:00:26.983]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testvPqnxK/42 with map size 1073741824
16:00:26 [16:00:26.986]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.006]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:cb96705d13e8f88d epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb96705d13e8f88d"}
16:00:27 [16:00:27.009]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.010]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:efd6a66fb8202ca8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:efd6a66fb8202ca8"}
16:00:27 [16:00:27.010]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRjEqQ0/1 with map size 1073741824
16:00:27 [16:00:27.012]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9c9146c6bee7ae69 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9c9146c6bee7ae69"}
16:00:27 [16:00:27.043]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.044]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b84a449ab05819e epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b84a449ab05819e"}
16:00:27 [16:00:27.044]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZopw8J/1 with map size 1073741824
16:00:27 [16:00:27.045]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.061]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3b84a449ab05819e epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3b84a449ab05819e"}
16:00:27 [16:00:27.063]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:faa6b031e5b5fab9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:faa6b031e5b5fab9"}
16:00:27 [16:00:27.064]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testq7NTXT/1 with map size 1073741824
16:00:27 [16:00:27.065]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.079]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:faa6b031e5b5fab9 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:faa6b031e5b5fab9"}
16:00:27 [16:00:27.079]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:faa6b031e5b5fab9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:faa6b031e5b5fab9"}
16:00:27 [16:00:27.084]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:faa6b031e5b5fab9 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:faa6b031e5b5fab9"}
16:00:27 [16:00:27.087]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd5746453a248a2b epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd5746453a248a2b"}
16:00:27 [16:00:27.088]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuXNDWJ/1 with map size 1073741824
16:00:27 [16:00:27.090]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.106]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.108]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.108]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e865cc847d68d81 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e865cc847d68d81"}
16:00:27 [16:00:27.109]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9EXm5M/1 with map size 1073741824
16:00:27 [16:00:27.110]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.125]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9e865cc847d68d81
{"provingJobId":"1:BASE_PARITY:9e865cc847d68d81"}
16:00:27 [16:00:27.126]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9e865cc847d68d81 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9e865cc847d68d81"}
16:00:27 [16:00:27.126]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9e865cc847d68d81 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:9e865cc847d68d81"}
16:00:27 [16:00:27.133]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fddc541bfed26a8b epochNumber=1
{"provingJobId":"1:BASE_PARITY:fddc541bfed26a8b"}
16:00:27 [16:00:27.134]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVuJ73Y/1 with map size 1073741824
16:00:27 [16:00:27.135]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.150]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:fddc541bfed26a8b
{"provingJobId":"1:BASE_PARITY:fddc541bfed26a8b"}
16:00:27 [16:00:27.150]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fddc541bfed26a8b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:fddc541bfed26a8b"}
16:00:27 [16:00:27.157]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.158]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fdb9568d5d53aaf1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fdb9568d5d53aaf1"}
16:00:27 [16:00:27.158]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVTLe5A/1 with map size 1073741824
16:00:27 [16:00:27.160]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.173]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fdb9568d5d53aaf1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fdb9568d5d53aaf1"}
16:00:27 [16:00:27.174]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fdb9568d5d53aaf1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fdb9568d5d53aaf1"}
16:00:27 [16:00:27.180]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.181]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef6512a760b40cfc epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef6512a760b40cfc"}
16:00:27 [16:00:27.181]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3zd47s/1 with map size 1073741824
16:00:27 [16:00:27.183]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.197]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ef6512a760b40cfc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ef6512a760b40cfc"}
16:00:27 [16:00:27.197]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ef6512a760b40cfc type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ef6512a760b40cfc"}
16:00:27 [16:00:27.203]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.204]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:493fa7ecff4fa9b4 epochNumber=0
{"provingJobId":"1:BASE_PARITY:493fa7ecff4fa9b4"}
16:00:27 [16:00:27.204]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testSdaCWR/0 with map size 1073741824
16:00:27 [16:00:27.206]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.219]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:493fa7ecff4fa9b4 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:493fa7ecff4fa9b4"}
16:00:27 [16:00:27.220]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:361cf41b64d03299 epochNumber=0
{"provingJobId":"1:BASE_PARITY:361cf41b64d03299"}
16:00:27 [16:00:27.220]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:493fa7ecff4fa9b4 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:493fa7ecff4fa9b4"}
16:00:27 [16:00:27.220]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c9cdc82533068a92 epochNumber=0
{"provingJobId":"1:BASE_PARITY:c9cdc82533068a92"}
16:00:27 [16:00:27.232]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.233]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.234]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6692d19ca3c8d385 epochNumber=2
{"provingJobId":"1:BASE_PARITY:6692d19ca3c8d385"}
16:00:27 [16:00:27.234]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testE7bjUD/2 with map size 1073741824
16:00:27 [16:00:27.235]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.250]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b8af93e11ce670c epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b8af93e11ce670c"}
16:00:27 [16:00:27.250]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testE7bjUD/1 with map size 1073741824
16:00:27 [16:00:27.253]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.268]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d68a377a7e562a0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9d68a377a7e562a0"}
16:00:27 [16:00:27.271]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJ3LHhm/1 with map size 1073741824
16:00:27 [16:00:27.272]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.288]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.288]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d80b3f799037f9e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d80b3f799037f9e"}
16:00:27 [16:00:27.289]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testV1F6tz/1 with map size 1073741824
16:00:27 [16:00:27.290]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.303]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df32a454db249852 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df32a454db249852"}
16:00:27 [16:00:27.308]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca78a5cade636855 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ca78a5cade636855"}
16:00:27 [16:00:27.309]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testV1F6tz/2 with map size 1073741824
16:00:27 [16:00:27.310]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.324]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0b7a5e2dd2a098a epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0b7a5e2dd2a098a"}
16:00:27 [16:00:27.331]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.332]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91db006c75a30faa epochNumber=1
{"provingJobId":"1:BASE_PARITY:91db006c75a30faa"}
16:00:27 [16:00:27.332]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0PUUvH/1 with map size 1073741824
16:00:27 [16:00:27.334]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.347]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c900692e7241e97 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c900692e7241e97"}
16:00:27 [16:00:27.352]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56f6e4bc833f5609 epochNumber=2
{"provingJobId":"1:BASE_PARITY:56f6e4bc833f5609"}
16:00:27 [16:00:27.352]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0PUUvH/2 with map size 1073741824
16:00:27 [16:00:27.354]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.368]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d784bda306c08e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d784bda306c08e1"}
16:00:27 [16:00:27.376]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.376]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b5513048f9722b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b5513048f9722b1"}
16:00:27 [16:00:27.377]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyR7kU7/1 with map size 1073741824
16:00:27 [16:00:27.378]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.392]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b89c85c91587113 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b89c85c91587113"}
16:00:27 [16:00:27.397]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a2997902c967818 epochNumber=2
{"provingJobId":"1:BASE_PARITY:7a2997902c967818"}
16:00:27 [16:00:27.397]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyR7kU7/2 with map size 1073741824
16:00:27 [16:00:27.399]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.412]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c26eaeed4b338a33 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c26eaeed4b338a33"}
16:00:27 [16:00:27.419]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.420]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aff023842224e9e8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aff023842224e9e8"}
16:00:27 [16:00:27.421]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdQbzco/1 with map size 1073741824
16:00:27 [16:00:27.422]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.435]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:639d1609abe8f70c epochNumber=1
{"provingJobId":"1:BASE_PARITY:639d1609abe8f70c"}
16:00:27 [16:00:27.440]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:aff023842224e9e8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:aff023842224e9e8"}
16:00:27 [16:00:27.447]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.448]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8366db56cc30ab8e epochNumber=1
{"provingJobId":"1:BASE_PARITY:8366db56cc30ab8e"}
16:00:27 [16:00:27.448]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJMG8D8/1 with map size 1073741824
16:00:27 [16:00:27.449]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.463]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:003ce972c51b77da epochNumber=1
{"provingJobId":"1:BASE_PARITY:003ce972c51b77da"}
16:00:27 [16:00:27.468]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8366db56cc30ab8e type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:8366db56cc30ab8e"}
16:00:27 [16:00:27.475]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.475]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6a3a1130100605f epochNumber=1
{"provingJobId":"1:BASE_PARITY:b6a3a1130100605f"}
16:00:27 [16:00:27.476]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testP0ZhE6/1 with map size 1073741824
16:00:27 [16:00:27.477]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.491]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c294302c0c9a5a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c294302c0c9a5a8"}
16:00:27 [16:00:27.496]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b6a3a1130100605f type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:b6a3a1130100605f"}
16:00:27 [16:00:27.499]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8cc5fb6eb2163431 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8cc5fb6eb2163431"}
16:00:27 [16:00:27.501]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4s0xmh/1 with map size 1073741824
16:00:27 [16:00:27.502]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.515]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:8cc5fb6eb2163431
{"provingJobId":"1:BASE_PARITY:8cc5fb6eb2163431"}
16:00:27 [16:00:27.516]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8cc5fb6eb2163431 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:8cc5fb6eb2163431"}
16:00:27 [16:00:27.521]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f34894089832833 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f34894089832833"}
16:00:27 [16:00:27.526]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8cc5fb6eb2163431 has already been completed
{"provingJobId":"1:BASE_PARITY:8cc5fb6eb2163431"}
16:00:27 [16:00:27.528]
INFO:
prover-client:proving-broker Proving Broker started
16:00:27 [16:00:27.529]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a599eae53b07ec37 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a599eae53b07ec37"}
16:00:27 [16:00:27.530]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQmSl8C/1 with map size 1073741824
16:00:27 [16:00:27.531]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:27 [16:00:27.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2a30be9cf25ad816 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2a30be9cf25ad816"}
16:00:27 [16:00:27.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQmSl8C/2 with map size 1073741824
16:00:27 [16:00:27.546]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:28 [16:00:28.611]
INFO:
prover-client:proving-broker Proving Broker started
16:00:28 [16:00:28.615]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a599eae53b07ec37 settled=false
{"provingJobId":"1:BASE_PARITY:a599eae53b07ec37","status":"pending"}
16:00:28 [16:00:28.616]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2a30be9cf25ad816 settled=false
{"provingJobId":"1:BASE_PARITY:2a30be9cf25ad816","status":"pending"}
16:00:28 [16:00:28.617]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a599eae53b07ec37 type=BASE_PARITY startedAt=1749830427559 older agent has taken job
{"provingJobId":"1:BASE_PARITY:a599eae53b07ec37"}
16:00:28 [16:00:28.617]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a599eae53b07ec37 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:a599eae53b07ec37"}
16:00:28 [16:00:28.620]
INFO:
prover-client:proving-broker Proving Broker started
16:00:28 [16:00:28.620]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71254a135f32664c epochNumber=1
{"provingJobId":"1:BASE_PARITY:71254a135f32664c"}
16:00:28 [16:00:28.621]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIk70BL/1 with map size 1073741824
16:00:28 [16:00:28.622]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:28 [16:00:28.636]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f84408d464ee124 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3f84408d464ee124"}
16:00:28 [16:00:28.636]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIk70BL/2 with map size 1073741824
16:00:28 [16:00:28.638]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:29 [16:00:29.651]
INFO:
prover-client:proving-broker Proving Broker started
16:00:29 [16:00:29.653]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:71254a135f32664c settled=false
{"provingJobId":"1:BASE_PARITY:71254a135f32664c","status":"pending"}
16:00:29 [16:00:29.654]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3f84408d464ee124 settled=false
{"provingJobId":"1:BASE_PARITY:3f84408d464ee124","status":"pending"}
16:00:29 [16:00:29.655]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:71254a135f32664c type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:71254a135f32664c"}
16:00:29 [16:00:29.658]
INFO:
prover-client:proving-broker Proving Broker started
16:00:29 [16:00:29.658]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c04ca972a83f761 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c04ca972a83f761"}
16:00:29 [16:00:29.659]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testb55KEz/1 with map size 1073741824
16:00:29 [16:00:29.660]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:29 [16:00:29.676]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba38152b598385d8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ba38152b598385d8"}
16:00:29 [16:00:29.676]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testb55KEz/2 with map size 1073741824
16:00:29 [16:00:29.678]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:30 [16:00:30.693]
INFO:
prover-client:proving-broker Proving Broker started
16:00:30 [16:00:30.696]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4c04ca972a83f761 settled=false
{"provingJobId":"1:BASE_PARITY:4c04ca972a83f761","status":"pending"}
16:00:30 [16:00:30.698]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ba38152b598385d8 settled=false
{"provingJobId":"1:BASE_PARITY:ba38152b598385d8","status":"pending"}
16:00:30 [16:00:30.699]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4c04ca972a83f761 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4c04ca972a83f761"}
16:00:30 [16:00:30.699]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4c04ca972a83f761 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4c04ca972a83f761"}
16:00:30 [16:00:30.709]
INFO:
prover-client:proving-broker Proving Broker started
16:00:30 [16:00:30.709]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f10fa1f05b121b84 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f10fa1f05b121b84"}
16:00:30 [16:00:30.710]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVzIAJX/1 with map size 1073741824
16:00:30 [16:00:30.711]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:30 [16:00:30.726]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0674f7652f4f554f epochNumber=2
{"provingJobId":"1:BASE_PARITY:0674f7652f4f554f"}
16:00:30 [16:00:30.727]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVzIAJX/2 with map size 1073741824
16:00:30 [16:00:30.728]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:30 [16:00:30.743]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f10fa1f05b121b84 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f10fa1f05b121b84"}
16:00:30 [16:00:30.748]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0674f7652f4f554f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0674f7652f4f554f"}
16:00:30 [16:00:30.756]
INFO:
prover-client:proving-broker Proving Broker started
16:00:30 [16:00:30.756]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:362241c846f30e93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:362241c846f30e93"}
16:00:30 [16:00:30.757]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testop6UJ2/1 with map size 1073741824
16:00:30 [16:00:30.758]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:30 [16:00:30.773]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48d93d6f3ef52aca epochNumber=2
{"provingJobId":"1:BASE_PARITY:48d93d6f3ef52aca"}
16:00:30 [16:00:30.774]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testop6UJ2/2 with map size 1073741824
16:00:30 [16:00:30.775]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:30 [16:00:30.789]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:362241c846f30e93 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:362241c846f30e93"}
16:00:30 [16:00:30.789]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:362241c846f30e93 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:362241c846f30e93"}
16:00:30 [16:00:30.794]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:48d93d6f3ef52aca type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:48d93d6f3ef52aca"}
16:00:30 [16:00:30.801]
INFO:
prover-client:proving-broker Proving Broker started
16:00:30 [16:00:30.802]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1d8ecf0e942fa75f err=test error
{"provingJoId":"1:BASE_PARITY:1d8ecf0e942fa75f"}
16:00:30 [16:00:30.804]
INFO:
prover-client:proving-broker Proving Broker started
16:00:30 [16:00:30.805]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:97a3032f44bfcf0c not found
{"provingJobId":"1:BASE_PARITY:97a3032f44bfcf0c"}
16:00:30 [16:00:30.807]
INFO:
prover-client:proving-broker Proving Broker started
16:00:30 [16:00:30.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:118a82332821659b epochNumber=1
{"provingJobId":"1:BASE_PARITY:118a82332821659b"}
16:00:30 [16:00:30.809]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyXUJDb/1 with map size 1073741824
16:00:30 [16:00:30.810]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:30 [16:00:30.826]
INFO:
prover-client:proving-broker Proving Broker started
16:00:30 [16:00:30.827]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48a544dabe9803a9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48a544dabe9803a9"}
16:00:30 [16:00:30.827]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1fOdqF/1 with map size 1073741824
16:00:30 [16:00:30.828]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:30 [16:00:30.953]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:48a544dabe9803a9 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:48a544dabe9803a9"}
16:00:30 [16:00:30.958]
INFO:
prover-client:proving-broker Proving Broker started
16:00:30 [16:00:30.958]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2fb0b0db464d8419 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2fb0b0db464d8419"}
16:00:30 [16:00:30.959]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTMSKgy/1 with map size 1073741824
16:00:30 [16:00:30.960]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.084]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2fb0b0db464d8419 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:2fb0b0db464d8419"}
16:00:31 [16:00:31.088]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:590a7a4cab55a48e epochNumber=10
{"provingJobId":"1:BASE_PARITY:590a7a4cab55a48e"}
16:00:31 [16:00:31.088]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testTMSKgy/10 with map size 1073741824
16:00:31 [16:00:31.090]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.110]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:31 [16:00:31.110]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:00:31 [16:00:31.231]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9a4fb6642ed3e94 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a9a4fb6642ed3e94"}
16:00:31 [16:00:31.233]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8eMa9D/1 with map size 1073741824
16:00:31 [16:00:31.236]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.406]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a9a4fb6642ed3e94 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a9a4fb6642ed3e94"}
16:00:31 [16:00:31.453]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:163edf8338f56ab7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:163edf8338f56ab7"}
16:00:31 [16:00:31.454]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testh0jqSK/1 with map size 1073741824
16:00:31 [16:00:31.455]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.471]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:163edf8338f56ab7 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:163edf8338f56ab7"}
16:00:31 [16:00:31.473]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.474]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d54828e583a04382 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d54828e583a04382"}
16:00:31 [16:00:31.474]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2yJUX6/1 with map size 1073741824
16:00:31 [16:00:31.476]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.491]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d54828e583a04382 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d54828e583a04382"}
16:00:31 [16:00:31.492]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d54828e583a04382 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:d54828e583a04382"}
16:00:31 [16:00:31.500]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:07f46602d6f82d3f epochNumber=1
{"provingJobId":"1:BASE_PARITY:07f46602d6f82d3f"}
16:00:31 [16:00:31.501]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testplnMIn/1 with map size 1073741824
16:00:31 [16:00:31.502]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.519]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:07f46602d6f82d3f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:07f46602d6f82d3f"}
16:00:31 [16:00:31.526]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd7e00206ccf6de8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cd7e00206ccf6de8"}
16:00:31 [16:00:31.528]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSuH3s4/1 with map size 1073741824
16:00:31 [16:00:31.530]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.544]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cd7e00206ccf6de8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:cd7e00206ccf6de8"}
16:00:31 [16:00:31.545]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59dc796427cde059 epochNumber=3
{"provingJobId":"1:BASE_PARITY:59dc796427cde059"}
16:00:31 [16:00:31.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testSuH3s4/3 with map size 1073741824
16:00:31 [16:00:31.547]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.576]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:31 [16:00:31.577]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:00:31 [16:00:31.585]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:cd7e00206ccf6de8 err=test error
{"provingJoId":"1:BASE_PARITY:cd7e00206ccf6de8"}
16:00:31 [16:00:31.588]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqPYVzf/1 with map size 1073741824
16:00:31 [16:00:31.590]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqPYVzf/2 with map size 1073741824
16:00:31 [16:00:31.608]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.625]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.626]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:92b1243b15bfc7b7 settled=false
{"provingJobId":"1:BASE_PARITY:92b1243b15bfc7b7","status":"pending"}
16:00:31 [16:00:31.628]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:16347b6de1252573 settled=false
{"provingJobId":"1:BASE_PARITY:16347b6de1252573","status":"pending"}
16:00:31 [16:00:31.632]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJKQaaJ/1 with map size 1073741824
16:00:31 [16:00:31.634]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.650]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testJKQaaJ/2 with map size 1073741824
16:00:31 [16:00:31.651]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.675]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.677]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2821fa90a083284f settled=true
{"provingJobId":"1:BASE_PARITY:2821fa90a083284f","status":"fulfilled"}
16:00:31 [16:00:31.679]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:7fd2aae5a87ca133 settled=true
{"provingJobId":"2:BASE_PARITY:7fd2aae5a87ca133","status":"fulfilled"}
16:00:31 [16:00:31.683]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMqglWH/1 with map size 1073741824
16:00:31 [16:00:31.684]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.705]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMqglWH/2 with map size 1073741824
16:00:31 [16:00:31.707]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.721]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.722]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:daac41f0c5fd8dc3 settled=true
{"provingJobId":"1:BASE_PARITY:daac41f0c5fd8dc3","status":"fulfilled"}
16:00:31 [16:00:31.725]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:912c5f992ac7e52f settled=false
{"provingJobId":"1:BASE_PARITY:912c5f992ac7e52f","status":"pending"}
16:00:31 [16:00:31.729]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.729]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3b0520d52ca22e2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3b0520d52ca22e2"}
16:00:31 [16:00:31.730]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjOaT2x/1 with map size 1073741824
16:00:31 [16:00:31.731]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.749]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.750]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44f8dfdac3895b5a epochNumber=1
{"provingJobId":"1:BASE_PARITY:44f8dfdac3895b5a"}
16:00:31 [16:00:31.750]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:44f8dfdac3895b5a: Error: db error: Error: db error
16:00:31 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:44f8dfdac3895b5a"}
16:00:31 [16:00:31.753]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.753]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ad831faf8c8b8390 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ad831faf8c8b8390"}
16:00:31 [16:00:31.754]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testw9VgEo/1 with map size 1073741824
16:00:31 [16:00:31.756]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.771]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ad831faf8c8b8390 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ad831faf8c8b8390"}
16:00:31 [16:00:31.771]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ad831faf8c8b8390 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ad831faf8c8b8390"}
16:00:31 [16:00:31.780]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.781]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4d151d59a319207 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4d151d59a319207"}
16:00:31 [16:00:31.781]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfgEjhs/1 with map size 1073741824
16:00:31 [16:00:31.783]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.797]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f4d151d59a319207 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f4d151d59a319207"}
16:00:31 [16:00:31.797]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f4d151d59a319207 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f4d151d59a319207"}
16:00:31 [16:00:31.798]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:f4d151d59a319207: Error: db error
16:00:31 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:f4d151d59a319207"}
16:00:31 [16:00:31.801]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:30a3df0a2d2d92c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:30a3df0a2d2d92c2"}
16:00:31 [16:00:31.802]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRcwnYY/1 with map size 1073741824
16:00:31 [16:00:31.804]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.818]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:30a3df0a2d2d92c2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:30a3df0a2d2d92c2"}
16:00:31 [16:00:31.818]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:30a3df0a2d2d92c2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:30a3df0a2d2d92c2"}
16:00:31 [16:00:31.826]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.826]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bde36ca4acad0cd8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bde36ca4acad0cd8"}
16:00:31 [16:00:31.827]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt0nJWF/1 with map size 1073741824
16:00:31 [16:00:31.828]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.842]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:bde36ca4acad0cd8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:bde36ca4acad0cd8"}
16:00:31 [16:00:31.842]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bde36ca4acad0cd8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:bde36ca4acad0cd8"}
16:00:31 [16:00:31.843]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:bde36ca4acad0cd8 jobErr=test error: Error: db error
16:00:31 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:bde36ca4acad0cd8"}
16:00:31 [16:00:31.845]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.846]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ecd462e650c387ed not found
{"provingJobId":"1:BASE_PARITY:ecd462e650c387ed"}
16:00:31 [16:00:31.848]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.849]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:62993c04528e1777 err=test error
{"provingJoId":"1:BASE_PARITY:62993c04528e1777"}
16:00:31 [16:00:31.852]
INFO:
prover-client:proving-broker Proving Broker started
16:00:31 [16:00:31.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4bea231653ebab41 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4bea231653ebab41"}
16:00:31 [16:00:31.863]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIM0IH2/1 with map size 1073741824
16:00:31 [16:00:31.864]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.879]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4bea231653ebab41 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4bea231653ebab41"}
16:00:31 [16:00:31.879]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4bea231653ebab41 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4bea231653ebab41"}
16:00:31 [16:00:31.894]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:e12714419785ce3c epochNumber=2
{"provingJobId":"2:BASE_PARITY:e12714419785ce3c"}
16:00:31 [16:00:31.895]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIM0IH2/2 with map size 1073741824
16:00:31 [16:00:31.896]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:31 [16:00:31.910]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:e12714419785ce3c type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:e12714419785ce3c"}
16:00:31 [16:00:31.911]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:e12714419785ce3c type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:e12714419785ce3c"}
16:00:31 [16:00:31.976]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:a4657b43c33d97b9 epochNumber=3
{"provingJobId":"3:BASE_PARITY:a4657b43c33d97b9"}
16:00:31 [16:00:31.977]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testIM0IH2/3 with map size 1073741824
16:00:31 [16:00:31.978]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:32 [16:00:32.004]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:32 [16:00:32.004]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
16:00:32 [16:00:32.053]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:0d2baa2f8a06c849 epochNumber=4
{"provingJobId":"4:BASE_PARITY:0d2baa2f8a06c849"}
16:00:32 [16:00:32.054]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testIM0IH2/4 with map size 1073741824
16:00:32 [16:00:32.055]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:32 [16:00:32.083]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:32 [16:00:32.083]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
16:00:32 [16:00:32.127]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:55eda24040396b1f epochNumber=5
{"provingJobId":"5:BASE_PARITY:55eda24040396b1f"}
16:00:32 [16:00:32.128]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testIM0IH2/5 with map size 1073741824
16:00:32 [16:00:32.130]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
16:00:32 [16:00:32.161]
INFO:
prover-client:proving-broker Cleaned up jobs=1
16:00:32 [16:00:32.161]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
16:00:32
PASS src/proving_broker/proving_broker.test.ts (
10.917 s)
16:00:32 ProvingBroker
16:00:32 Producer API
16:00:32
✓ refuses stale jobs (33 ms)
16:00:32
✓ enqueues jobs (4 ms)
16:00:32
✓ ignores duplicate jobs (4 ms)
16:00:32
✓ reports correct status when enqueuing repeat jobs (4 ms)
16:00:32
✓ throws an error in case of duplicate job IDs (22 ms)
16:00:32
✓ returns not-found status for non-existing jobs (1 ms)
16:00:32
✓ cancels jobs in queue (3 ms)
16:00:32
✓ cancels jobs in-progress (2 ms)
16:00:32
✓ returns job result if successful (2 ms)
16:00:32
✓ returns job error if failed (2 ms)
16:00:32
✓ correctly returns job status for concurrent writes (2 ms)
16:00:32
✓ refuses stale jobs (28 ms)
16:00:32
✓ enqueues jobs (34 ms)
16:00:32
✓ ignores duplicate jobs (20 ms)
16:00:32
✓ reports correct status when enqueuing repeat jobs (23 ms)
16:00:32
✓ throws an error in case of duplicate job IDs (19 ms)
16:00:32
✓ returns not-found status for non-existing jobs (2 ms)
16:00:32
✓ cancels jobs in queue (25 ms)
16:00:32
✓ cancels jobs in-progress (24 ms)
16:00:32
✓ returns job result if successful (23 ms)
16:00:32
✓ returns job error if failed (23 ms)
16:00:32
✓ correctly returns job status for concurrent writes (28 ms)
16:00:32 Consumer API
16:00:32
✓ returns undefined if no jobs are available (1 ms)
16:00:32
✓ returns jobs in priority order (2 ms)
16:00:32
✓ returns undefined if no jobs are available for the given allowList (1 ms)
16:00:32
✓ returns a job if it is in the allowList (2 ms)
16:00:32
✓ returns the most important job if it is in the allowList (2 ms)
16:00:32
✓ returns any job if filter is empty (1 ms)
16:00:32
✓ returns a new job when reporting job success (2 ms)
16:00:32
✓ returns a new job when reporting permanent error (1 ms)
16:00:32
✓ returns a new job when reporting retry-able error (1 ms)
16:00:32
✓ returns a new job when reporting progress if current one is cancelled (2 ms)
16:00:32
✓ returns a new job if job is already in progress elsewhere (1054 ms)
16:00:32
✓ avoids sending the same job to a new agent after a restart (1003 ms)
16:00:32
✓ avoids sending a completed job to a new agent after a restart (1004 ms)
16:00:32
✓ tracks job result if in progress (3 ms)
16:00:32
✓ tracks job result even if job is in queue (2 ms)
16:00:32
✓ ignores reported job error if unknown job (1 ms)
16:00:32
✓ ignores job result if unknown job (1 ms)
16:00:32
✓ returns undefined if no jobs are available (1 ms)
16:00:32
✓ returns jobs in priority order (34 ms)
16:00:32
✓ returns undefined if no jobs are available for the given allowList (19 ms)
16:00:32
✓ returns a job if it is in the allowList (43 ms)
16:00:32
✓ returns the most important job if it is in the allowList (44 ms)
16:00:32
✓ returns any job if filter is empty (43 ms)
16:00:32
✓ returns a new job when reporting job success (28 ms)
16:00:32
✓ returns a new job when reporting permanent error (27 ms)
16:00:32
✓ returns a new job when reporting retry-able error (24 ms)
16:00:32
✓ returns a new job when reporting progress if current one is cancelled (29 ms)
16:00:32
✓ returns a new job if job is already in progress elsewhere (1091 ms)
16:00:32
✓ avoids sending the same job to a new agent after a restart (1037 ms)
16:00:32
✓ avoids sending a completed job to a new agent after a restart (1050 ms)
16:00:32
✓ tracks job result if in progress (46 ms)
16:00:32
✓ tracks job result even if job is in queue (45 ms)
16:00:32
✓ ignores reported job error if unknown job (3 ms)
16:00:32
✓ ignores job result if unknown job (3 ms)
16:00:32 Timeouts
16:00:32
✓ tracks in progress jobs (1 ms)
16:00:32
✓ re-enqueues jobs that time out (108 ms)
16:00:32
✓ cancel stale jobs that time out (234 ms)
16:00:32
✓ keeps the jobs in progress while it is alive (202 ms)
16:00:32
✓ tracks in progress jobs (19 ms)
16:00:32
✓ re-enqueues jobs that time out (131 ms)
16:00:32
✓ cancel stale jobs that time out (274 ms)
16:00:32
✓ keeps the jobs in progress while it is alive (221 ms)
16:00:32 Retries
16:00:32
✓ retries jobs (2 ms)
16:00:32
✓ retries up to a maximum number of times (2 ms)
16:00:32
✓ passing retry=false does not retry (2 ms)
16:00:32
✓ does not retry if job is stale (27 ms)
16:00:32
✓ retries jobs (20 ms)
16:00:32
✓ retries up to a maximum number of times (26 ms)
16:00:32
✓ passing retry=false does not retry (27 ms)
16:00:32
✓ does not retry if job is stale (62 ms)
16:00:32 Database management
16:00:32
✓ re-enqueues proof requests on start (2 ms)
16:00:32
✓ restores proof results on start (1 ms)
16:00:32
✓ only re-enqueues unfinished jobs (1 ms)
16:00:32
✓ saves job when enqueued (3 ms)
16:00:32
✓ does not retain job if database fails to save (2 ms)
16:00:32
✓ saves job result (2 ms)
16:00:32
✓ saves result even if database fails to save (2 ms)
16:00:32
✓ saves job error (2 ms)
16:00:32
✓ saves job error even if database fails to save (2 ms)
16:00:32
✓ does not save job result if job is unknown (2 ms)
16:00:32
✓ does not save job error if job is unknown (1 ms)
16:00:32
✓ cleans up old jobs periodically (256 ms)
16:00:32
✓ re-enqueues proof requests on start (43 ms)
16:00:32
✓ restores proof results on start (51 ms)
16:00:32
✓ only re-enqueues unfinished jobs (46 ms)
16:00:32
✓ saves job when enqueued (20 ms)
16:00:32
✓ does not retain job if database fails to save (3 ms)
16:00:32
✓ saves job result (27 ms)
16:00:32
✓ saves result even if database fails to save (22 ms)
16:00:32
✓ saves job error (24 ms)
16:00:32
✓ saves job error even if database fails to save (20 ms)
16:00:32
✓ does not save job result if job is unknown (3 ms)
16:00:32
✓ does not save job error if job is unknown (3 ms)
16:00:32
✓ cleans up old jobs periodically (347 ms)
16:00:32
16:00:32
Test Suites: 1 passed, 1 total
16:00:32
Tests: 96 passed, 96 total
16:00:32
Snapshots: 0 total
16:00:32
Time: 10.987 s
16:00:32
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
16:00:32
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?