Parent Log:
http://ci.aztec-labs.com/84a46107c517e2c5
Command: 71ec8227227d28e6 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/a45107e7f95b675cb2768b6bcb06483b511141f4
Env: REF_NAME=gh-readonly-queue/next/pr-15018-2731d9a8aebf63d05db1a786296abdf897c49490 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Thu Jun 12 17:44:14 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15018_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
17:44:19 [17:44:19.126]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.135]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba600fbe0abfea6a epochNumber=42
{"provingJobId":"1:BASE_PARITY:ba600fbe0abfea6a"}
17:44:19 [17:44:19.139]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:a45aa46fafde78e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a45aa46fafde78e1"}
17:44:19 [17:44:19.178]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.179]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1aabca04179e1a66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1aabca04179e1a66"}
17:44:19 [17:44:19.180]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9ff5354ac99a6c4d epochNumber=1
{"provingJobId":"1:BASE_PARITY:9ff5354ac99a6c4d"}
17:44:19 [17:44:19.182]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.182]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:abeb44ab4630317f epochNumber=1
{"provingJobId":"1:BASE_PARITY:abeb44ab4630317f"}
17:44:19 [17:44:19.183]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:abeb44ab4630317f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:abeb44ab4630317f"}
17:44:19 [17:44:19.186]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.187]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:949b8d41031a4bf5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:949b8d41031a4bf5"}
17:44:19 [17:44:19.188]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:949b8d41031a4bf5 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:949b8d41031a4bf5"}
17:44:19 [17:44:19.189]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:949b8d41031a4bf5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:949b8d41031a4bf5"}
17:44:19 [17:44:19.189]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:949b8d41031a4bf5 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:949b8d41031a4bf5"}
17:44:19 [17:44:19.191]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.191]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b93ec9bf1cd23bf9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b93ec9bf1cd23bf9"}
17:44:19 [17:44:19.217]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.219]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.220]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe415d087046b29e epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe415d087046b29e"}
17:44:19 [17:44:19.221]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:fe415d087046b29e
{"provingJobId":"1:BASE_PARITY:fe415d087046b29e"}
17:44:19 [17:44:19.221]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fe415d087046b29e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fe415d087046b29e"}
17:44:19 [17:44:19.221]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fe415d087046b29e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:fe415d087046b29e"}
17:44:19 [17:44:19.223]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.223]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca2dc2aa27214d7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca2dc2aa27214d7d"}
17:44:19 [17:44:19.224]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ca2dc2aa27214d7d
{"provingJobId":"1:BASE_PARITY:ca2dc2aa27214d7d"}
17:44:19 [17:44:19.224]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ca2dc2aa27214d7d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ca2dc2aa27214d7d"}
17:44:19 [17:44:19.225]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.225]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b25f9cb26d34f6ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:b25f9cb26d34f6ca"}
17:44:19 [17:44:19.226]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b25f9cb26d34f6ca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b25f9cb26d34f6ca"}
17:44:19 [17:44:19.226]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b25f9cb26d34f6ca type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b25f9cb26d34f6ca"}
17:44:19 [17:44:19.227]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.227]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:da13f95f77b09d71 epochNumber=1
{"provingJobId":"1:BASE_PARITY:da13f95f77b09d71"}
17:44:19 [17:44:19.228]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:da13f95f77b09d71 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:da13f95f77b09d71"}
17:44:19 [17:44:19.228]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:da13f95f77b09d71 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:da13f95f77b09d71"}
17:44:19 [17:44:19.229]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.229]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1451563d8271b235 epochNumber=0
{"provingJobId":"1:BASE_PARITY:1451563d8271b235"}
17:44:19 [17:44:19.229]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1451563d8271b235 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1451563d8271b235"}
17:44:19 [17:44:19.229]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e08a75aa1e2500d epochNumber=0
{"provingJobId":"1:BASE_PARITY:6e08a75aa1e2500d"}
17:44:19 [17:44:19.230]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1451563d8271b235 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1451563d8271b235"}
17:44:19 [17:44:19.230]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a29eb6402fec47dc epochNumber=0
{"provingJobId":"1:BASE_PARITY:a29eb6402fec47dc"}
17:44:19 [17:44:19.231]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.231]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09aae7529097ca5c epochNumber=2
{"provingJobId":"1:BASE_PARITY:09aae7529097ca5c"}
17:44:19 [17:44:19.232]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f0c93ab897475a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f0c93ab897475a7"}
17:44:19 [17:44:19.233]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.233]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1889de8137f8f6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1889de8137f8f6a"}
17:44:19 [17:44:19.234]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.234]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c2bf1f4d0814984 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c2bf1f4d0814984"}
17:44:19 [17:44:19.234]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9a8cacf7ba0c92d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9a8cacf7ba0c92d6"}
17:44:19 [17:44:19.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cbbee739ef130e4a epochNumber=2
{"provingJobId":"1:BASE_PARITY:cbbee739ef130e4a"}
17:44:19 [17:44:19.235]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57a30d10ca81a33d epochNumber=1
{"provingJobId":"1:BASE_PARITY:57a30d10ca81a33d"}
17:44:19 [17:44:19.236]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a5099a9a2bb6457 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a5099a9a2bb6457"}
17:44:19 [17:44:19.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de25c796814853a2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de25c796814853a2"}
17:44:19 [17:44:19.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e94391db24603dcf epochNumber=2
{"provingJobId":"1:BASE_PARITY:e94391db24603dcf"}
17:44:19 [17:44:19.237]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c2be8b62da2c219 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c2be8b62da2c219"}
17:44:19 [17:44:19.237]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee5a65db5a8dd3bd epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee5a65db5a8dd3bd"}
17:44:19 [17:44:19.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d76bb1e43ffb0ac3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d76bb1e43ffb0ac3"}
17:44:19 [17:44:19.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68c05a8fa3d91dd3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:68c05a8fa3d91dd3"}
17:44:19 [17:44:19.238]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:102237ca5eeb3240 epochNumber=1
{"provingJobId":"1:BASE_PARITY:102237ca5eeb3240"}
17:44:19 [17:44:19.239]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.239]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0d980f2d6c07220 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0d980f2d6c07220"}
17:44:19 [17:44:19.240]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34922f4c104ab6b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:34922f4c104ab6b8"}
17:44:19 [17:44:19.240]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e0d980f2d6c07220 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e0d980f2d6c07220"}
17:44:19 [17:44:19.241]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.241]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:29e4193e42e0e8e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:29e4193e42e0e8e6"}
17:44:19 [17:44:19.241]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95ca7874b09b613c epochNumber=1
{"provingJobId":"1:BASE_PARITY:95ca7874b09b613c"}
17:44:19 [17:44:19.242]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:29e4193e42e0e8e6 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:29e4193e42e0e8e6"}
17:44:19 [17:44:19.242]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.243]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:51d9875512629d86 epochNumber=1
{"provingJobId":"1:BASE_PARITY:51d9875512629d86"}
17:44:19 [17:44:19.243]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c35007becefb432a epochNumber=1
{"provingJobId":"1:BASE_PARITY:c35007becefb432a"}
17:44:19 [17:44:19.243]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:51d9875512629d86 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:51d9875512629d86"}
17:44:19 [17:44:19.244]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.244]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e8d0e8acf5d02344 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e8d0e8acf5d02344"}
17:44:19 [17:44:19.245]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e8d0e8acf5d02344
{"provingJobId":"1:BASE_PARITY:e8d0e8acf5d02344"}
17:44:19 [17:44:19.245]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e8d0e8acf5d02344 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:e8d0e8acf5d02344"}
17:44:19 [17:44:19.245]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6d00ae87159ab63 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6d00ae87159ab63"}
17:44:19 [17:44:19.253]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e8d0e8acf5d02344 has already been completed
{"provingJobId":"1:BASE_PARITY:e8d0e8acf5d02344"}
17:44:19 [17:44:19.254]
INFO:
prover-client:proving-broker Proving Broker started
17:44:19 [17:44:19.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:383d8d1b4ed506c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:383d8d1b4ed506c7"}
17:44:19 [17:44:19.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5abf46c8004091ac epochNumber=2
{"provingJobId":"1:BASE_PARITY:5abf46c8004091ac"}
17:44:20 [17:44:20.308]
INFO:
prover-client:proving-broker Proving Broker started
17:44:20 [17:44:20.309]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:383d8d1b4ed506c7 settled=false
{"provingJobId":"1:BASE_PARITY:383d8d1b4ed506c7","status":"pending"}
17:44:20 [17:44:20.309]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5abf46c8004091ac settled=false
{"provingJobId":"1:BASE_PARITY:5abf46c8004091ac","status":"pending"}
17:44:20 [17:44:20.310]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:383d8d1b4ed506c7 type=BASE_PARITY startedAt=1749750259256 older agent has taken job
{"provingJobId":"1:BASE_PARITY:383d8d1b4ed506c7"}
17:44:20 [17:44:20.310]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:383d8d1b4ed506c7 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:383d8d1b4ed506c7"}
17:44:20 [17:44:20.311]
INFO:
prover-client:proving-broker Proving Broker started
17:44:20 [17:44:20.312]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8800ebc82bcc0f93 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8800ebc82bcc0f93"}
17:44:20 [17:44:20.312]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:42fe4bafdd52164a epochNumber=2
{"provingJobId":"1:BASE_PARITY:42fe4bafdd52164a"}
17:44:21 [17:44:21.314]
INFO:
prover-client:proving-broker Proving Broker started
17:44:21 [17:44:21.316]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8800ebc82bcc0f93 settled=false
{"provingJobId":"1:BASE_PARITY:8800ebc82bcc0f93","status":"pending"}
17:44:21 [17:44:21.317]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:42fe4bafdd52164a settled=false
{"provingJobId":"1:BASE_PARITY:42fe4bafdd52164a","status":"pending"}
17:44:21 [17:44:21.318]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8800ebc82bcc0f93 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:8800ebc82bcc0f93"}
17:44:21 [17:44:21.322]
INFO:
prover-client:proving-broker Proving Broker started
17:44:21 [17:44:21.323]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:86e906c4bb60fd1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:86e906c4bb60fd1f"}
17:44:21 [17:44:21.327]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14d0f05cc393f7fd epochNumber=2
{"provingJobId":"1:BASE_PARITY:14d0f05cc393f7fd"}
17:44:22 [17:44:22.328]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.329]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:86e906c4bb60fd1f settled=false
{"provingJobId":"1:BASE_PARITY:86e906c4bb60fd1f","status":"pending"}
17:44:22 [17:44:22.329]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:14d0f05cc393f7fd settled=false
{"provingJobId":"1:BASE_PARITY:14d0f05cc393f7fd","status":"pending"}
17:44:22 [17:44:22.330]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:86e906c4bb60fd1f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:86e906c4bb60fd1f"}
17:44:22 [17:44:22.330]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:86e906c4bb60fd1f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:86e906c4bb60fd1f"}
17:44:22 [17:44:22.332]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.333]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:15d27cf40c1c0dc0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:15d27cf40c1c0dc0"}
17:44:22 [17:44:22.333]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:04111ba08b74c3d8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:04111ba08b74c3d8"}
17:44:22 [17:44:22.334]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:15d27cf40c1c0dc0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:15d27cf40c1c0dc0"}
17:44:22 [17:44:22.335]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:04111ba08b74c3d8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:04111ba08b74c3d8"}
17:44:22 [17:44:22.336]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.337]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d4e207731b6c6a9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d4e207731b6c6a9"}
17:44:22 [17:44:22.337]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88cf14b1ab9a6ba2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:88cf14b1ab9a6ba2"}
17:44:22 [17:44:22.338]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3d4e207731b6c6a9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3d4e207731b6c6a9"}
17:44:22 [17:44:22.338]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3d4e207731b6c6a9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3d4e207731b6c6a9"}
17:44:22 [17:44:22.339]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:88cf14b1ab9a6ba2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:88cf14b1ab9a6ba2"}
17:44:22 [17:44:22.340]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.341]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:82228fc887bf06c6 err=test error
{"provingJoId":"1:BASE_PARITY:82228fc887bf06c6"}
17:44:22 [17:44:22.342]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.342]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:13b95dc6453322a1 not found
{"provingJobId":"1:BASE_PARITY:13b95dc6453322a1"}
17:44:22 [17:44:22.344]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.344]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e8cec66c09af3af epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e8cec66c09af3af"}
17:44:22 [17:44:22.346]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8dd4a65682e64cb0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8dd4a65682e64cb0"}
17:44:22 [17:44:22.447]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8dd4a65682e64cb0 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8dd4a65682e64cb0"}
17:44:22 [17:44:22.449]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.450]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31ca262c4c3d6e07 epochNumber=1
{"provingJobId":"1:BASE_PARITY:31ca262c4c3d6e07"}
17:44:22 [17:44:22.550]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:31ca262c4c3d6e07 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:31ca262c4c3d6e07"}
17:44:22 [17:44:22.552]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cea5536831881f99 epochNumber=10
{"provingJobId":"1:BASE_PARITY:cea5536831881f99"}
17:44:22 [17:44:22.576]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:22 [17:44:22.680]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.681]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1d3047eb9372378e epochNumber=1
{"provingJobId":"1:BASE_PARITY:1d3047eb9372378e"}
17:44:22 [17:44:22.832]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1d3047eb9372378e timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1d3047eb9372378e"}
17:44:22 [17:44:22.884]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.885]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:114b0528484da74e epochNumber=1
{"provingJobId":"1:BASE_PARITY:114b0528484da74e"}
17:44:22 [17:44:22.885]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:114b0528484da74e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:114b0528484da74e"}
17:44:22 [17:44:22.887]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.887]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc6bb69d9c8fba64 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc6bb69d9c8fba64"}
17:44:22 [17:44:22.888]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:dc6bb69d9c8fba64 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:dc6bb69d9c8fba64"}
17:44:22 [17:44:22.889]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:dc6bb69d9c8fba64 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:dc6bb69d9c8fba64"}
17:44:22 [17:44:22.889]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.890]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89341fd9e257b901 epochNumber=1
{"provingJobId":"1:BASE_PARITY:89341fd9e257b901"}
17:44:22 [17:44:22.890]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:89341fd9e257b901 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:89341fd9e257b901"}
17:44:22 [17:44:22.891]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.891]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c3118c7006810c1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c3118c7006810c1"}
17:44:22 [17:44:22.892]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2c3118c7006810c1 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:2c3118c7006810c1"}
17:44:22 [17:44:22.892]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f4560ebb1b80fd4 epochNumber=3
{"provingJobId":"1:BASE_PARITY:9f4560ebb1b80fd4"}
17:44:22 [17:44:22.916]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:22 [17:44:22.918]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2c3118c7006810c1 err=test error
{"provingJoId":"1:BASE_PARITY:2c3118c7006810c1"}
17:44:22 [17:44:22.920]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.920]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c6e54cd1ff604cae settled=false
{"provingJobId":"1:BASE_PARITY:c6e54cd1ff604cae","status":"pending"}
17:44:22 [17:44:22.920]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:92ebe1d984109119 settled=false
{"provingJobId":"1:BASE_PARITY:92ebe1d984109119","status":"pending"}
17:44:22 [17:44:22.922]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.922]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9af76eebcada1ec4 settled=true
{"provingJobId":"1:BASE_PARITY:9af76eebcada1ec4","status":"fulfilled"}
17:44:22 [17:44:22.923]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:f78d01278da3bb71 settled=true
{"provingJobId":"2:BASE_PARITY:f78d01278da3bb71","status":"fulfilled"}
17:44:22 [17:44:22.924]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.924]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5f8aa3f0dfc2dc37 settled=true
{"provingJobId":"1:BASE_PARITY:5f8aa3f0dfc2dc37","status":"fulfilled"}
17:44:22 [17:44:22.924]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0873be72f914d23d settled=false
{"provingJobId":"1:BASE_PARITY:0873be72f914d23d","status":"pending"}
17:44:22 [17:44:22.926]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.928]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6bc7eede4870fbf4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6bc7eede4870fbf4"}
17:44:22 [17:44:22.931]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.931]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a275319f74f7aa26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a275319f74f7aa26"}
17:44:22 [17:44:22.933]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:a275319f74f7aa26: Error: db error: Error: db error
17:44:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:a275319f74f7aa26"}
17:44:22 [17:44:22.934]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.935]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1c746a77ed256fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:d1c746a77ed256fd"}
17:44:22 [17:44:22.935]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d1c746a77ed256fd type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:d1c746a77ed256fd"}
17:44:22 [17:44:22.935]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d1c746a77ed256fd type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d1c746a77ed256fd"}
17:44:22 [17:44:22.937]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.937]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e5399059f33ec96b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5399059f33ec96b"}
17:44:22 [17:44:22.938]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e5399059f33ec96b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e5399059f33ec96b"}
17:44:22 [17:44:22.938]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e5399059f33ec96b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e5399059f33ec96b"}
17:44:22 [17:44:22.939]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:e5399059f33ec96b: Error: db error
17:44:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:e5399059f33ec96b"}
17:44:22 [17:44:22.940]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a78ddbc6f4148cd7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a78ddbc6f4148cd7"}
17:44:22 [17:44:22.940]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a78ddbc6f4148cd7 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a78ddbc6f4148cd7"}
17:44:22 [17:44:22.941]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a78ddbc6f4148cd7 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a78ddbc6f4148cd7"}
17:44:22 [17:44:22.942]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.943]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b5b960009120997 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b5b960009120997"}
17:44:22 [17:44:22.943]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6b5b960009120997 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6b5b960009120997"}
17:44:22 [17:44:22.943]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6b5b960009120997 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6b5b960009120997"}
17:44:22 [17:44:22.944]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:6b5b960009120997 jobErr=test error: Error: db error
17:44:22 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:6b5b960009120997"}
17:44:22 [17:44:22.945]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.946]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:12c1215b5e2e0f8a not found
{"provingJobId":"1:BASE_PARITY:12c1215b5e2e0f8a"}
17:44:22 [17:44:22.947]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.948]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4e8eec9be6ee7042 err=test error
{"provingJoId":"1:BASE_PARITY:4e8eec9be6ee7042"}
17:44:22 [17:44:22.949]
INFO:
prover-client:proving-broker Proving Broker started
17:44:22 [17:44:22.960]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81b0f30046cc094d epochNumber=1
{"provingJobId":"1:BASE_PARITY:81b0f30046cc094d"}
17:44:22 [17:44:22.961]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:81b0f30046cc094d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:81b0f30046cc094d"}
17:44:22 [17:44:22.961]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:81b0f30046cc094d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:81b0f30046cc094d"}
17:44:22 [17:44:22.972]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:bdccf8cb14b7b847 epochNumber=2
{"provingJobId":"2:BASE_PARITY:bdccf8cb14b7b847"}
17:44:22 [17:44:22.973]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:bdccf8cb14b7b847 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:bdccf8cb14b7b847"}
17:44:22 [17:44:22.973]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:bdccf8cb14b7b847 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:bdccf8cb14b7b847"}
17:44:23 [17:44:23.034]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:837c843987ae01f7 epochNumber=3
{"provingJobId":"3:BASE_PARITY:837c843987ae01f7"}
17:44:23 [17:44:23.050]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:23 [17:44:23.096]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:f5a91bf7ef2a5c92 epochNumber=4
{"provingJobId":"4:BASE_PARITY:f5a91bf7ef2a5c92"}
17:44:23 [17:44:23.102]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:23 [17:44:23.158]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:72072a66d454dcca epochNumber=5
{"provingJobId":"5:BASE_PARITY:72072a66d454dcca"}
17:44:23 [17:44:23.180]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:23 [17:44:23.214]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.215]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:303dd9f370f2302f epochNumber=42
{"provingJobId":"1:BASE_PARITY:303dd9f370f2302f"}
17:44:23 [17:44:23.219]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-test5K4cat/42 with map size 1073741824
17:44:23 [17:44:23.223]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.263]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:e25386e572cb8118 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e25386e572cb8118"}
17:44:23 [17:44:23.268]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.268]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:18f24a18e7b08f81 epochNumber=1
{"provingJobId":"1:BASE_PARITY:18f24a18e7b08f81"}
17:44:23 [17:44:23.270]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testp3A352/1 with map size 1073741824
17:44:23 [17:44:23.273]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.292]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5a006d2bf0c1ce23 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5a006d2bf0c1ce23"}
17:44:23 [17:44:23.306]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.307]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b127b7e523409710 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b127b7e523409710"}
17:44:23 [17:44:23.308]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testi0qWDT/1 with map size 1073741824
17:44:23 [17:44:23.313]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.336]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b127b7e523409710 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b127b7e523409710"}
17:44:23 [17:44:23.343]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.344]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9e416efff799c29 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9e416efff799c29"}
17:44:23 [17:44:23.345]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3tDtYn/1 with map size 1073741824
17:44:23 [17:44:23.348]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.365]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b9e416efff799c29 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b9e416efff799c29"}
17:44:23 [17:44:23.366]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b9e416efff799c29 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b9e416efff799c29"}
17:44:23 [17:44:23.372]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b9e416efff799c29 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b9e416efff799c29"}
17:44:23 [17:44:23.375]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.376]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0b8d776072ef22f epochNumber=1
{"provingJobId":"1:BASE_PARITY:c0b8d776072ef22f"}
17:44:23 [17:44:23.377]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testraFYlz/1 with map size 1073741824
17:44:23 [17:44:23.378]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.399]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.401]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.402]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ff54c1d9900d9369 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ff54c1d9900d9369"}
17:44:23 [17:44:23.402]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeAXxay/1 with map size 1073741824
17:44:23 [17:44:23.405]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.422]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ff54c1d9900d9369
{"provingJobId":"1:BASE_PARITY:ff54c1d9900d9369"}
17:44:23 [17:44:23.423]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ff54c1d9900d9369 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ff54c1d9900d9369"}
17:44:23 [17:44:23.423]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ff54c1d9900d9369 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:ff54c1d9900d9369"}
17:44:23 [17:44:23.433]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.434]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88d1493d2077cd2e epochNumber=1
{"provingJobId":"1:BASE_PARITY:88d1493d2077cd2e"}
17:44:23 [17:44:23.434]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTaSNQZ/1 with map size 1073741824
17:44:23 [17:44:23.441]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.460]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:88d1493d2077cd2e
{"provingJobId":"1:BASE_PARITY:88d1493d2077cd2e"}
17:44:23 [17:44:23.461]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:88d1493d2077cd2e type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:88d1493d2077cd2e"}
17:44:23 [17:44:23.469]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.470]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7c02fc8d67064593 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7c02fc8d67064593"}
17:44:23 [17:44:23.471]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testigvBMQ/1 with map size 1073741824
17:44:23 [17:44:23.473]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.489]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7c02fc8d67064593 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:7c02fc8d67064593"}
17:44:23 [17:44:23.490]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7c02fc8d67064593 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7c02fc8d67064593"}
17:44:23 [17:44:23.499]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:016e29c3547c44c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:016e29c3547c44c3"}
17:44:23 [17:44:23.501]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7FsUkk/1 with map size 1073741824
17:44:23 [17:44:23.504]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.521]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:016e29c3547c44c3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:016e29c3547c44c3"}
17:44:23 [17:44:23.521]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:016e29c3547c44c3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:016e29c3547c44c3"}
17:44:23 [17:44:23.533]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.535]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d59be87e496e702 epochNumber=0
{"provingJobId":"1:BASE_PARITY:3d59be87e496e702"}
17:44:23 [17:44:23.536]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test4tnDaQ/0 with map size 1073741824
17:44:23 [17:44:23.539]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.557]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3d59be87e496e702 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3d59be87e496e702"}
17:44:23 [17:44:23.558]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68a5ffa460e25792 epochNumber=0
{"provingJobId":"1:BASE_PARITY:68a5ffa460e25792"}
17:44:23 [17:44:23.559]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3d59be87e496e702 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3d59be87e496e702"}
17:44:23 [17:44:23.559]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a0ff4b511df9cfda epochNumber=0
{"provingJobId":"1:BASE_PARITY:a0ff4b511df9cfda"}
17:44:23 [17:44:23.575]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.578]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.578]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:739eeb70edb8ffe3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:739eeb70edb8ffe3"}
17:44:23 [17:44:23.579]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testmycIVx/2 with map size 1073741824
17:44:23 [17:44:23.581]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.597]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:45f3eee0d7fe23c8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:45f3eee0d7fe23c8"}
17:44:23 [17:44:23.598]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmycIVx/1 with map size 1073741824
17:44:23 [17:44:23.600]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.621]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.621]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6bc57636ae357dc7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6bc57636ae357dc7"}
17:44:23 [17:44:23.622]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testALUb7r/1 with map size 1073741824
17:44:23 [17:44:23.624]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.650]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.651]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c9ac6cea266ab73 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c9ac6cea266ab73"}
17:44:23 [17:44:23.653]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testD4URH7/1 with map size 1073741824
17:44:23 [17:44:23.656]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.682]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e979acef2875d49a epochNumber=1
{"provingJobId":"1:BASE_PARITY:e979acef2875d49a"}
17:44:23 [17:44:23.691]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f22a98784a8e989b epochNumber=2
{"provingJobId":"1:BASE_PARITY:f22a98784a8e989b"}
17:44:23 [17:44:23.693]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testD4URH7/2 with map size 1073741824
17:44:23 [17:44:23.696]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.723]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b8f8dfce3bf7bd8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b8f8dfce3bf7bd8"}
17:44:23 [17:44:23.744]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.745]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:617839bde64e40a3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:617839bde64e40a3"}
17:44:23 [17:44:23.747]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testorylZE/1 with map size 1073741824
17:44:23 [17:44:23.754]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.777]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8aff94c126726bd3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8aff94c126726bd3"}
17:44:23 [17:44:23.785]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:292429c6e819d3a3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:292429c6e819d3a3"}
17:44:23 [17:44:23.786]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testorylZE/2 with map size 1073741824
17:44:23 [17:44:23.789]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.810]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d9c4452b045dc26 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d9c4452b045dc26"}
17:44:23 [17:44:23.824]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.825]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f1b8e7b67da6f10 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f1b8e7b67da6f10"}
17:44:23 [17:44:23.825]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testidlcCA/1 with map size 1073741824
17:44:23 [17:44:23.830]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:815712ccc4d65388 epochNumber=1
{"provingJobId":"1:BASE_PARITY:815712ccc4d65388"}
17:44:23 [17:44:23.853]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f20c41b0f105ef38 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f20c41b0f105ef38"}
17:44:23 [17:44:23.854]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testidlcCA/2 with map size 1073741824
17:44:23 [17:44:23.856]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.877]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7d938c213ecb108d epochNumber=1
{"provingJobId":"1:BASE_PARITY:7d938c213ecb108d"}
17:44:23 [17:44:23.890]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.890]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af7ee6aedbe0f903 epochNumber=1
{"provingJobId":"1:BASE_PARITY:af7ee6aedbe0f903"}
17:44:23 [17:44:23.891]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxHDsOJ/1 with map size 1073741824
17:44:23 [17:44:23.894]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.917]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22f37a1579b62ca3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:22f37a1579b62ca3"}
17:44:23 [17:44:23.923]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:af7ee6aedbe0f903 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:af7ee6aedbe0f903"}
17:44:23 [17:44:23.938]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3e1076d3ff575ea8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3e1076d3ff575ea8"}
17:44:23 [17:44:23.942]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfjLJuW/1 with map size 1073741824
17:44:23 [17:44:23.945]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:23 [17:44:23.963]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6fe48eca4b3aa9a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6fe48eca4b3aa9a7"}
17:44:23 [17:44:23.969]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3e1076d3ff575ea8 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:3e1076d3ff575ea8"}
17:44:23 [17:44:23.979]
INFO:
prover-client:proving-broker Proving Broker started
17:44:23 [17:44:23.980]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa7adda5f809cea8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa7adda5f809cea8"}
17:44:23 [17:44:23.983]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1z9rsu/1 with map size 1073741824
17:44:23 [17:44:23.986]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:24 [17:44:24.010]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7a1660bf88b6f151 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7a1660bf88b6f151"}
17:44:24 [17:44:24.016]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:aa7adda5f809cea8 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:aa7adda5f809cea8"}
17:44:24 [17:44:24.022]
INFO:
prover-client:proving-broker Proving Broker started
17:44:24 [17:44:24.023]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:023f442d62c55b11 epochNumber=1
{"provingJobId":"1:BASE_PARITY:023f442d62c55b11"}
17:44:24 [17:44:24.023]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testD8JpDU/1 with map size 1073741824
17:44:24 [17:44:24.025]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:24 [17:44:24.049]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:023f442d62c55b11
{"provingJobId":"1:BASE_PARITY:023f442d62c55b11"}
17:44:24 [17:44:24.050]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:023f442d62c55b11 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:023f442d62c55b11"}
17:44:24 [17:44:24.061]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:043d51f17d530787 epochNumber=1
{"provingJobId":"1:BASE_PARITY:043d51f17d530787"}
17:44:24 [17:44:24.068]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:023f442d62c55b11 has already been completed
{"provingJobId":"1:BASE_PARITY:023f442d62c55b11"}
17:44:24 [17:44:24.074]
INFO:
prover-client:proving-broker Proving Broker started
17:44:24 [17:44:24.076]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0619958879a58a7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c0619958879a58a7"}
17:44:24 [17:44:24.077]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTaI56S/1 with map size 1073741824
17:44:24 [17:44:24.080]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:24 [17:44:24.099]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c788b627ffc55574 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c788b627ffc55574"}
17:44:24 [17:44:24.100]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTaI56S/2 with map size 1073741824
17:44:24 [17:44:24.103]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:25 [17:44:25.176]
INFO:
prover-client:proving-broker Proving Broker started
17:44:25 [17:44:25.184]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c0619958879a58a7 settled=false
{"provingJobId":"1:BASE_PARITY:c0619958879a58a7","status":"pending"}
17:44:25 [17:44:25.185]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c788b627ffc55574 settled=false
{"provingJobId":"1:BASE_PARITY:c788b627ffc55574","status":"pending"}
17:44:25 [17:44:25.186]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c0619958879a58a7 type=BASE_PARITY startedAt=1749750264124 older agent has taken job
{"provingJobId":"1:BASE_PARITY:c0619958879a58a7"}
17:44:25 [17:44:25.187]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c0619958879a58a7 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:c0619958879a58a7"}
17:44:25 [17:44:25.191]
INFO:
prover-client:proving-broker Proving Broker started
17:44:25 [17:44:25.192]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c17f4cbb9dffc92 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c17f4cbb9dffc92"}
17:44:25 [17:44:25.193]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test64UXDg/1 with map size 1073741824
17:44:25 [17:44:25.194]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:25 [17:44:25.213]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:febc7ad9a0ef482e epochNumber=2
{"provingJobId":"1:BASE_PARITY:febc7ad9a0ef482e"}
17:44:25 [17:44:25.214]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test64UXDg/2 with map size 1073741824
17:44:25 [17:44:25.217]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:26 [17:44:26.235]
INFO:
prover-client:proving-broker Proving Broker started
17:44:26 [17:44:26.238]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8c17f4cbb9dffc92 settled=false
{"provingJobId":"1:BASE_PARITY:8c17f4cbb9dffc92","status":"pending"}
17:44:26 [17:44:26.240]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:febc7ad9a0ef482e settled=false
{"provingJobId":"1:BASE_PARITY:febc7ad9a0ef482e","status":"pending"}
17:44:26 [17:44:26.241]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8c17f4cbb9dffc92 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:8c17f4cbb9dffc92"}
17:44:26 [17:44:26.245]
INFO:
prover-client:proving-broker Proving Broker started
17:44:26 [17:44:26.247]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f511801045171c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f511801045171c2"}
17:44:26 [17:44:26.248]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYFTAcQ/1 with map size 1073741824
17:44:26 [17:44:26.249]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:26 [17:44:26.269]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9d220d2c30b31494 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9d220d2c30b31494"}
17:44:26 [17:44:26.270]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYFTAcQ/2 with map size 1073741824
17:44:26 [17:44:26.274]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:27 [17:44:27.292]
INFO:
prover-client:proving-broker Proving Broker started
17:44:27 [17:44:27.296]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4f511801045171c2 settled=false
{"provingJobId":"1:BASE_PARITY:4f511801045171c2","status":"pending"}
17:44:27 [17:44:27.298]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9d220d2c30b31494 settled=false
{"provingJobId":"1:BASE_PARITY:9d220d2c30b31494","status":"pending"}
17:44:27 [17:44:27.299]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4f511801045171c2 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4f511801045171c2"}
17:44:27 [17:44:27.299]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4f511801045171c2 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4f511801045171c2"}
17:44:27 [17:44:27.311]
INFO:
prover-client:proving-broker Proving Broker started
17:44:27 [17:44:27.311]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:af3abd1f30ac06bb epochNumber=1
{"provingJobId":"1:BASE_PARITY:af3abd1f30ac06bb"}
17:44:27 [17:44:27.312]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test424ogr/1 with map size 1073741824
17:44:27 [17:44:27.315]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:27 [17:44:27.350]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31325a60b3e73e6d epochNumber=2
{"provingJobId":"1:BASE_PARITY:31325a60b3e73e6d"}
17:44:27 [17:44:27.353]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test424ogr/2 with map size 1073741824
17:44:27 [17:44:27.357]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:27 [17:44:27.381]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:af3abd1f30ac06bb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:af3abd1f30ac06bb"}
17:44:27 [17:44:27.388]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:31325a60b3e73e6d type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:31325a60b3e73e6d"}
17:44:27 [17:44:27.398]
INFO:
prover-client:proving-broker Proving Broker started
17:44:27 [17:44:27.399]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:351acfa7f3f45890 epochNumber=1
{"provingJobId":"1:BASE_PARITY:351acfa7f3f45890"}
17:44:27 [17:44:27.400]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZX5sJz/1 with map size 1073741824
17:44:27 [17:44:27.404]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:27 [17:44:27.423]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef2f22d0be9bc431 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ef2f22d0be9bc431"}
17:44:27 [17:44:27.425]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZX5sJz/2 with map size 1073741824
17:44:27 [17:44:27.431]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:27 [17:44:27.453]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:351acfa7f3f45890 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:351acfa7f3f45890"}
17:44:27 [17:44:27.454]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:351acfa7f3f45890 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:351acfa7f3f45890"}
17:44:27 [17:44:27.462]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ef2f22d0be9bc431 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ef2f22d0be9bc431"}
17:44:27 [17:44:27.477]
INFO:
prover-client:proving-broker Proving Broker started
17:44:27 [17:44:27.478]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:cd165db827f17218 err=test error
{"provingJoId":"1:BASE_PARITY:cd165db827f17218"}
17:44:27 [17:44:27.483]
INFO:
prover-client:proving-broker Proving Broker started
17:44:27 [17:44:27.484]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e2b97d6046238e5f not found
{"provingJobId":"1:BASE_PARITY:e2b97d6046238e5f"}
17:44:27 [17:44:27.486]
INFO:
prover-client:proving-broker Proving Broker started
17:44:27 [17:44:27.487]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b045ae1ef74febc epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b045ae1ef74febc"}
17:44:27 [17:44:27.488]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testELUdIm/1 with map size 1073741824
17:44:27 [17:44:27.490]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:27 [17:44:27.513]
INFO:
prover-client:proving-broker Proving Broker started
17:44:27 [17:44:27.514]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a19b86b0e77d5152 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a19b86b0e77d5152"}
17:44:27 [17:44:27.514]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkd7agu/1 with map size 1073741824
17:44:27 [17:44:27.519]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:27 [17:44:27.667]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a19b86b0e77d5152 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a19b86b0e77d5152"}
17:44:27 [17:44:27.671]
INFO:
prover-client:proving-broker Proving Broker started
17:44:27 [17:44:27.672]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c18cca89874703c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c18cca89874703c0"}
17:44:27 [17:44:27.673]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9dnCGp/1 with map size 1073741824
17:44:27 [17:44:27.675]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:27 [17:44:27.801]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c18cca89874703c0 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c18cca89874703c0"}
17:44:27 [17:44:27.808]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bab24900a721f7c3 epochNumber=10
{"provingJobId":"1:BASE_PARITY:bab24900a721f7c3"}
17:44:27 [17:44:27.808]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test9dnCGp/10 with map size 1073741824
17:44:27 [17:44:27.810]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:27 [17:44:27.857]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:27 [17:44:27.857]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:44:27 [17:44:27.960]
INFO:
prover-client:proving-broker Proving Broker started
17:44:27 [17:44:27.961]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ada132fc794f03a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ada132fc794f03a8"}
17:44:27 [17:44:27.962]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsgmyXL/1 with map size 1073741824
17:44:27 [17:44:27.964]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.144]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ada132fc794f03a8 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:ada132fc794f03a8"}
17:44:28 [17:44:28.194]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.196]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:96d46843d60c74de epochNumber=1
{"provingJobId":"1:BASE_PARITY:96d46843d60c74de"}
17:44:28 [17:44:28.197]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbCUV5Q/1 with map size 1073741824
17:44:28 [17:44:28.200]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.224]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:96d46843d60c74de type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:96d46843d60c74de"}
17:44:28 [17:44:28.234]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.236]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:257f0b28fe8804a8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:257f0b28fe8804a8"}
17:44:28 [17:44:28.237]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfA87sa/1 with map size 1073741824
17:44:28 [17:44:28.243]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.269]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:257f0b28fe8804a8 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:257f0b28fe8804a8"}
17:44:28 [17:44:28.271]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:257f0b28fe8804a8 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:257f0b28fe8804a8"}
17:44:28 [17:44:28.283]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a25e47725f3d3f52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a25e47725f3d3f52"}
17:44:28 [17:44:28.288]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf33uyl/1 with map size 1073741824
17:44:28 [17:44:28.292]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.310]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a25e47725f3d3f52 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a25e47725f3d3f52"}
17:44:28 [17:44:28.321]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.322]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf5df04eff62886d epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf5df04eff62886d"}
17:44:28 [17:44:28.323]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testT726Gr/1 with map size 1073741824
17:44:28 [17:44:28.331]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.355]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cf5df04eff62886d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:cf5df04eff62886d"}
17:44:28 [17:44:28.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:00117090575adfd0 epochNumber=3
{"provingJobId":"1:BASE_PARITY:00117090575adfd0"}
17:44:28 [17:44:28.357]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testT726Gr/3 with map size 1073741824
17:44:28 [17:44:28.359]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.397]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:28 [17:44:28.398]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:44:28 [17:44:28.404]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:cf5df04eff62886d err=test error
{"provingJoId":"1:BASE_PARITY:cf5df04eff62886d"}
17:44:28 [17:44:28.408]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBPzdu6/1 with map size 1073741824
17:44:28 [17:44:28.409]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.437]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testBPzdu6/2 with map size 1073741824
17:44:28 [17:44:28.443]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.464]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.466]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c77df73ef9b99fb0 settled=false
{"provingJobId":"1:BASE_PARITY:c77df73ef9b99fb0","status":"pending"}
17:44:28 [17:44:28.468]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:460354d40757894f settled=false
{"provingJobId":"1:BASE_PARITY:460354d40757894f","status":"pending"}
17:44:28 [17:44:28.475]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYAmeHM/1 with map size 1073741824
17:44:28 [17:44:28.479]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.497]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYAmeHM/2 with map size 1073741824
17:44:28 [17:44:28.500]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.534]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.543]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0949ddd4a20cd3c8 settled=true
{"provingJobId":"1:BASE_PARITY:0949ddd4a20cd3c8","status":"fulfilled"}
17:44:28 [17:44:28.548]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:817a2c29f39810a4 settled=true
{"provingJobId":"2:BASE_PARITY:817a2c29f39810a4","status":"fulfilled"}
17:44:28 [17:44:28.565]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDDol8b/1 with map size 1073741824
17:44:28 [17:44:28.569]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.594]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDDol8b/2 with map size 1073741824
17:44:28 [17:44:28.598]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.617]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.619]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4eea51962236d4d1 settled=true
{"provingJobId":"1:BASE_PARITY:4eea51962236d4d1","status":"fulfilled"}
17:44:28 [17:44:28.623]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:05f710be0f14535e settled=false
{"provingJobId":"1:BASE_PARITY:05f710be0f14535e","status":"pending"}
17:44:28 [17:44:28.634]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.635]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1f793c1d7a0a161a epochNumber=1
{"provingJobId":"1:BASE_PARITY:1f793c1d7a0a161a"}
17:44:28 [17:44:28.656]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaptrzD/1 with map size 1073741824
17:44:28 [17:44:28.661]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.700]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.702]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0651d3e656b3145e epochNumber=1
{"provingJobId":"1:BASE_PARITY:0651d3e656b3145e"}
17:44:28 [17:44:28.708]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:0651d3e656b3145e: Error: db error: Error: db error
17:44:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:0651d3e656b3145e"}
17:44:28 [17:44:28.724]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.725]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d8df0c8208d67f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d8df0c8208d67f0"}
17:44:28 [17:44:28.735]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdnbrno/1 with map size 1073741824
17:44:28 [17:44:28.753]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.775]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3d8df0c8208d67f0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3d8df0c8208d67f0"}
17:44:28 [17:44:28.776]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3d8df0c8208d67f0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3d8df0c8208d67f0"}
17:44:28 [17:44:28.797]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14328bf0c0855389 epochNumber=1
{"provingJobId":"1:BASE_PARITY:14328bf0c0855389"}
17:44:28 [17:44:28.802]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgn5WoJ/1 with map size 1073741824
17:44:28 [17:44:28.806]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.827]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:14328bf0c0855389 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:14328bf0c0855389"}
17:44:28 [17:44:28.828]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:14328bf0c0855389 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:14328bf0c0855389"}
17:44:28 [17:44:28.829]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:14328bf0c0855389: Error: db error
17:44:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:14328bf0c0855389"}
17:44:28 [17:44:28.847]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.848]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:543a70dd0722ead5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:543a70dd0722ead5"}
17:44:28 [17:44:28.850]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdBLYaI/1 with map size 1073741824
17:44:28 [17:44:28.855]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.876]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:543a70dd0722ead5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:543a70dd0722ead5"}
17:44:28 [17:44:28.877]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:543a70dd0722ead5 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:543a70dd0722ead5"}
17:44:28 [17:44:28.891]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.893]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a83237ccfae6a294 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a83237ccfae6a294"}
17:44:28 [17:44:28.895]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgOw9j3/1 with map size 1073741824
17:44:28 [17:44:28.899]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.922]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a83237ccfae6a294 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a83237ccfae6a294"}
17:44:28 [17:44:28.923]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a83237ccfae6a294 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a83237ccfae6a294"}
17:44:28 [17:44:28.924]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a83237ccfae6a294 jobErr=test error: Error: db error
17:44:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:a83237ccfae6a294"}
17:44:28 [17:44:28.943]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.946]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2828cf58852656d4 not found
{"provingJobId":"1:BASE_PARITY:2828cf58852656d4"}
17:44:28 [17:44:28.955]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.956]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2ceee2cc48b4b8eb err=test error
{"provingJoId":"1:BASE_PARITY:2ceee2cc48b4b8eb"}
17:44:28 [17:44:28.959]
INFO:
prover-client:proving-broker Proving Broker started
17:44:28 [17:44:28.969]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:82c20fad0772ae20 epochNumber=1
{"provingJobId":"1:BASE_PARITY:82c20fad0772ae20"}
17:44:28 [17:44:28.970]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWMN0Mt/1 with map size 1073741824
17:44:28 [17:44:28.972]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:28 [17:44:28.989]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:82c20fad0772ae20 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:82c20fad0772ae20"}
17:44:28 [17:44:28.989]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:82c20fad0772ae20 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:82c20fad0772ae20"}
17:44:29 [17:44:29.006]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:a44165e347c7bab6 epochNumber=2
{"provingJobId":"2:BASE_PARITY:a44165e347c7bab6"}
17:44:29 [17:44:29.007]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWMN0Mt/2 with map size 1073741824
17:44:29 [17:44:29.012]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:29 [17:44:29.035]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:a44165e347c7bab6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:a44165e347c7bab6"}
17:44:29 [17:44:29.036]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:a44165e347c7bab6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:a44165e347c7bab6"}
17:44:29 [17:44:29.107]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:27903b99f1cc4447 epochNumber=3
{"provingJobId":"3:BASE_PARITY:27903b99f1cc4447"}
17:44:29 [17:44:29.108]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testWMN0Mt/3 with map size 1073741824
17:44:29 [17:44:29.111]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:29 [17:44:29.140]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:29 [17:44:29.141]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
17:44:29 [17:44:29.201]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:d0cba0946f975b5c epochNumber=4
{"provingJobId":"4:BASE_PARITY:d0cba0946f975b5c"}
17:44:29 [17:44:29.203]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testWMN0Mt/4 with map size 1073741824
17:44:29 [17:44:29.206]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:29 [17:44:29.225]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:29 [17:44:29.226]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
17:44:29 [17:44:29.287]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:8b8646bc5f15d351 epochNumber=5
{"provingJobId":"5:BASE_PARITY:8b8646bc5f15d351"}
17:44:29 [17:44:29.289]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testWMN0Mt/5 with map size 1073741824
17:44:29 [17:44:29.291]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
17:44:29 [17:44:29.315]
INFO:
prover-client:proving-broker Cleaned up jobs=1
17:44:29 [17:44:29.316]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
17:44:29
PASS src/proving_broker/proving_broker.test.ts (
13.74 s)
17:44:29 ProvingBroker
17:44:29 Producer API
17:44:29
✓ refuses stale jobs (49 ms)
17:44:29
✓ enqueues jobs (9 ms)
17:44:29
✓ ignores duplicate jobs (4 ms)
17:44:29
✓ reports correct status when enqueuing repeat jobs (4 ms)
17:44:29
✓ throws an error in case of duplicate job IDs (26 ms)
17:44:29
✓ returns not-found status for non-existing jobs (1 ms)
17:44:29
✓ cancels jobs in queue (3 ms)
17:44:29
✓ cancels jobs in-progress (2 ms)
17:44:29
✓ returns job result if successful (2 ms)
17:44:29
✓ returns job error if failed (1 ms)
17:44:29
✓ correctly returns job status for concurrent writes (2 ms)
17:44:29
✓ refuses stale jobs (55 ms)
17:44:29
✓ enqueues jobs (36 ms)
17:44:29
✓ ignores duplicate jobs (36 ms)
17:44:29
✓ reports correct status when enqueuing repeat jobs (34 ms)
17:44:29
✓ throws an error in case of duplicate job IDs (23 ms)
17:44:29
✓ returns not-found status for non-existing jobs (3 ms)
17:44:29
✓ cancels jobs in queue (31 ms)
17:44:29
✓ cancels jobs in-progress (36 ms)
17:44:29
✓ returns job result if successful (30 ms)
17:44:29
✓ returns job error if failed (32 ms)
17:44:29
✓ correctly returns job status for concurrent writes (43 ms)
17:44:29 Consumer API
17:44:29
✓ returns undefined if no jobs are available (1 ms)
17:44:29
✓ returns jobs in priority order (2 ms)
17:44:29
✓ returns undefined if no jobs are available for the given allowList (1 ms)
17:44:29
✓ returns a job if it is in the allowList (1 ms)
17:44:29
✓ returns the most important job if it is in the allowList (2 ms)
17:44:29
✓ returns any job if filter is empty (2 ms)
17:44:29
✓ returns a new job when reporting job success (2 ms)
17:44:29
✓ returns a new job when reporting permanent error (1 ms)
17:44:29
✓ returns a new job when reporting retry-able error (1 ms)
17:44:29
✓ returns a new job when reporting progress if current one is cancelled (10 ms)
17:44:29
✓ returns a new job if job is already in progress elsewhere (1057 ms)
17:44:29
✓ avoids sending the same job to a new agent after a restart (1009 ms)
17:44:29
✓ avoids sending a completed job to a new agent after a restart (1010 ms)
17:44:29
✓ tracks job result if in progress (3 ms)
17:44:29
✓ tracks job result even if job is in queue (3 ms)
17:44:29
✓ ignores reported job error if unknown job (2 ms)
17:44:29
✓ ignores job result if unknown job (2 ms)
17:44:29
✓ returns undefined if no jobs are available (3 ms)
17:44:29
✓ returns jobs in priority order (43 ms)
17:44:29
✓ returns undefined if no jobs are available for the given allowList (24 ms)
17:44:29
✓ returns a job if it is in the allowList (97 ms)
17:44:29
✓ returns the most important job if it is in the allowList (80 ms)
17:44:29
✓ returns any job if filter is empty (66 ms)
17:44:29
✓ returns a new job when reporting job success (47 ms)
17:44:29
✓ returns a new job when reporting permanent error (41 ms)
17:44:29
✓ returns a new job when reporting retry-able error (42 ms)
17:44:29
✓ returns a new job when reporting progress if current one is cancelled (51 ms)
17:44:29
✓ returns a new job if job is already in progress elsewhere (1118 ms)
17:44:29
✓ avoids sending the same job to a new agent after a restart (1054 ms)
17:44:29
✓ avoids sending a completed job to a new agent after a restart (1066 ms)
17:44:29
✓ tracks job result if in progress (87 ms)
17:44:29
✓ tracks job result even if job is in queue (77 ms)
17:44:29
✓ ignores reported job error if unknown job (6 ms)
17:44:29
✓ ignores job result if unknown job (5 ms)
17:44:29 Timeouts
17:44:29
✓ tracks in progress jobs (2 ms)
17:44:29
✓ re-enqueues jobs that time out (103 ms)
17:44:29
✓ cancel stale jobs that time out (230 ms)
17:44:29
✓ keeps the jobs in progress while it is alive (204 ms)
17:44:29
✓ tracks in progress jobs (25 ms)
17:44:29
✓ re-enqueues jobs that time out (159 ms)
17:44:29
✓ cancel stale jobs that time out (289 ms)
17:44:29
✓ keeps the jobs in progress while it is alive (231 ms)
17:44:29 Retries
17:44:29
✓ retries jobs (3 ms)
17:44:29
✓ retries up to a maximum number of times (3 ms)
17:44:29
✓ passing retry=false does not retry (2 ms)
17:44:29
✓ does not retry if job is stale (28 ms)
17:44:29
✓ retries jobs (35 ms)
17:44:29
✓ retries up to a maximum number of times (53 ms)
17:44:29
✓ passing retry=false does not retry (38 ms)
17:44:29
✓ does not retry if job is stale (86 ms)
17:44:29 Database management
17:44:29
✓ re-enqueues proof requests on start (2 ms)
17:44:29
✓ restores proof results on start (1 ms)
17:44:29
✓ only re-enqueues unfinished jobs (2 ms)
17:44:29
✓ saves job when enqueued (4 ms)
17:44:29
✓ does not retain job if database fails to save (4 ms)
17:44:29
✓ saves job result (2 ms)
17:44:29
✓ saves result even if database fails to save (3 ms)
17:44:29
✓ saves job error (2 ms)
17:44:29
✓ saves job error even if database fails to save (2 ms)
17:44:29
✓ does not save job result if job is unknown (2 ms)
17:44:29
✓ does not save job error if job is unknown (1 ms)
17:44:29
✓ cleans up old jobs periodically (261 ms)
17:44:29
✓ re-enqueues proof requests on start (66 ms)
17:44:29
✓ restores proof results on start (82 ms)
17:44:29
✓ only re-enqueues unfinished jobs (71 ms)
17:44:29
✓ saves job when enqueued (61 ms)
17:44:29
✓ does not retain job if database fails to save (26 ms)
17:44:29
✓ saves job result (75 ms)
17:44:29
✓ saves result even if database fails to save (48 ms)
17:44:29
✓ saves job error (46 ms)
17:44:29
✓ saves job error even if database fails to save (43 ms)
17:44:29
✓ does not save job result if job is unknown (16 ms)
17:44:29
✓ does not save job error if job is unknown (8 ms)
17:44:29
✓ cleans up old jobs periodically (406 ms)
17:44:29
17:44:29
Test Suites: 1 passed, 1 total
17:44:29
Tests: 96 passed, 96 total
17:44:29
Snapshots: 0 total
17:44:29
Time: 13.82 s
17:44:29
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
17:44:29
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?