Parent Log:
http://ci.aztec-labs.com/ddad17d0c78b0b72
Command: f44e1b4f382283e1 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/be1fa8aacb33d330cf496352bb1446439e34c51a
Env: REF_NAME=gh-readonly-queue/next/pr-15162-7d1abcb337009939ee47da8445b5fee80f39fb58 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 20 10:04:25 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15162_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
10:04:30 [10:04:30.014]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.022]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3078dd4728f4302a epochNumber=42
{"provingJobId":"1:BASE_PARITY:3078dd4728f4302a"}
10:04:30 [10:04:30.025]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:a07477bfdbf87124 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a07477bfdbf87124"}
10:04:30 [10:04:30.059]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.060]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc16973a2406f9d0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc16973a2406f9d0"}
10:04:30 [10:04:30.061]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:559a5d6a9d22b547 epochNumber=1
{"provingJobId":"1:BASE_PARITY:559a5d6a9d22b547"}
10:04:30 [10:04:30.063]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.064]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e84a5f3204bffc3b epochNumber=1
{"provingJobId":"1:BASE_PARITY:e84a5f3204bffc3b"}
10:04:30 [10:04:30.065]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e84a5f3204bffc3b epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e84a5f3204bffc3b"}
10:04:30 [10:04:30.067]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.068]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd7a30330ff84b16 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd7a30330ff84b16"}
10:04:30 [10:04:30.069]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fd7a30330ff84b16 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fd7a30330ff84b16"}
10:04:30 [10:04:30.069]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fd7a30330ff84b16 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fd7a30330ff84b16"}
10:04:30 [10:04:30.070]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fd7a30330ff84b16 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fd7a30330ff84b16"}
10:04:30 [10:04:30.071]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.071]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1ba295406f6bc2e epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1ba295406f6bc2e"}
10:04:30 [10:04:30.097]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.098]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.098]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:626f9207a121011a epochNumber=1
{"provingJobId":"1:BASE_PARITY:626f9207a121011a"}
10:04:30 [10:04:30.099]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:626f9207a121011a
{"provingJobId":"1:BASE_PARITY:626f9207a121011a"}
10:04:30 [10:04:30.099]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:626f9207a121011a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:626f9207a121011a"}
10:04:30 [10:04:30.100]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:626f9207a121011a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:626f9207a121011a"}
10:04:30 [10:04:30.101]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.101]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ec37fb771787a72 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3ec37fb771787a72"}
10:04:30 [10:04:30.102]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:3ec37fb771787a72
{"provingJobId":"1:BASE_PARITY:3ec37fb771787a72"}
10:04:30 [10:04:30.102]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3ec37fb771787a72 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:3ec37fb771787a72"}
10:04:30 [10:04:30.108]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.109]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31516c5a0e50ee52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:31516c5a0e50ee52"}
10:04:30 [10:04:30.109]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:31516c5a0e50ee52 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:31516c5a0e50ee52"}
10:04:30 [10:04:30.109]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:31516c5a0e50ee52 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:31516c5a0e50ee52"}
10:04:30 [10:04:30.111]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.111]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1576d783b0bb3e5c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1576d783b0bb3e5c"}
10:04:30 [10:04:30.111]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1576d783b0bb3e5c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1576d783b0bb3e5c"}
10:04:30 [10:04:30.112]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1576d783b0bb3e5c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1576d783b0bb3e5c"}
10:04:30 [10:04:30.112]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.113]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c3dc49f9163146bc epochNumber=0
{"provingJobId":"1:BASE_PARITY:c3dc49f9163146bc"}
10:04:30 [10:04:30.113]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c3dc49f9163146bc epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c3dc49f9163146bc"}
10:04:30 [10:04:30.113]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a662a5433b727a7f epochNumber=0
{"provingJobId":"1:BASE_PARITY:a662a5433b727a7f"}
10:04:30 [10:04:30.114]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c3dc49f9163146bc epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:c3dc49f9163146bc"}
10:04:30 [10:04:30.114]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e121527f57bcf2e7 epochNumber=0
{"provingJobId":"1:BASE_PARITY:e121527f57bcf2e7"}
10:04:30 [10:04:30.115]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.116]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.116]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85f4dd978cc7b0d5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:85f4dd978cc7b0d5"}
10:04:30 [10:04:30.116]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fe5ba466bbf9de9b epochNumber=1
{"provingJobId":"1:BASE_PARITY:fe5ba466bbf9de9b"}
10:04:30 [10:04:30.117]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.118]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e517f535aff29473 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e517f535aff29473"}
10:04:30 [10:04:30.118]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68ff057035737b22 epochNumber=1
{"provingJobId":"1:BASE_PARITY:68ff057035737b22"}
10:04:30 [10:04:30.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:40effab580f1916e epochNumber=1
{"provingJobId":"1:BASE_PARITY:40effab580f1916e"}
10:04:30 [10:04:30.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b5d6423e34dcf1b epochNumber=2
{"provingJobId":"1:BASE_PARITY:0b5d6423e34dcf1b"}
10:04:30 [10:04:30.120]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2ab6660d281343e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2ab6660d281343e"}
10:04:30 [10:04:30.121]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.121]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0d50e1c6faaa5f92 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0d50e1c6faaa5f92"}
10:04:30 [10:04:30.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14d5bab57bac7b34 epochNumber=1
{"provingJobId":"1:BASE_PARITY:14d5bab57bac7b34"}
10:04:30 [10:04:30.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4942d06b48330bc epochNumber=2
{"provingJobId":"1:BASE_PARITY:c4942d06b48330bc"}
10:04:30 [10:04:30.122]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8093121fa014f86e epochNumber=1
{"provingJobId":"1:BASE_PARITY:8093121fa014f86e"}
10:04:30 [10:04:30.123]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.123]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:17b3393ece693255 epochNumber=1
{"provingJobId":"1:BASE_PARITY:17b3393ece693255"}
10:04:30 [10:04:30.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4099db08e392866e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4099db08e392866e"}
10:04:30 [10:04:30.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b6a06094dde31ce epochNumber=2
{"provingJobId":"1:BASE_PARITY:6b6a06094dde31ce"}
10:04:30 [10:04:30.124]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c27e626173716de0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c27e626173716de0"}
10:04:30 [10:04:30.125]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.125]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4aa4496631d1728a epochNumber=1
{"provingJobId":"1:BASE_PARITY:4aa4496631d1728a"}
10:04:30 [10:04:30.125]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:63031cc46ebfe556 epochNumber=1
{"provingJobId":"1:BASE_PARITY:63031cc46ebfe556"}
10:04:30 [10:04:30.126]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4aa4496631d1728a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4aa4496631d1728a"}
10:04:30 [10:04:30.127]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ae2ff3bd08152a9c epochNumber=1
{"provingJobId":"1:BASE_PARITY:ae2ff3bd08152a9c"}
10:04:30 [10:04:30.127]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b06f77fbf753a4a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b06f77fbf753a4a6"}
10:04:30 [10:04:30.127]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ae2ff3bd08152a9c type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:ae2ff3bd08152a9c"}
10:04:30 [10:04:30.128]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.129]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb871cd332d85b37 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb871cd332d85b37"}
10:04:30 [10:04:30.129]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c703a6a06eabf919 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c703a6a06eabf919"}
10:04:30 [10:04:30.129]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cb871cd332d85b37 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:cb871cd332d85b37"}
10:04:30 [10:04:30.130]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.131]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fceea3cc83a27a6f epochNumber=1
{"provingJobId":"1:BASE_PARITY:fceea3cc83a27a6f"}
10:04:30 [10:04:30.131]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:fceea3cc83a27a6f
{"provingJobId":"1:BASE_PARITY:fceea3cc83a27a6f"}
10:04:30 [10:04:30.132]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fceea3cc83a27a6f type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:fceea3cc83a27a6f"}
10:04:30 [10:04:30.132]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:73cda01edf1ad087 epochNumber=1
{"provingJobId":"1:BASE_PARITY:73cda01edf1ad087"}
10:04:30 [10:04:30.132]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fceea3cc83a27a6f has already been completed
{"provingJobId":"1:BASE_PARITY:fceea3cc83a27a6f"}
10:04:30 [10:04:30.133]
INFO:
prover-client:proving-broker Proving Broker started
10:04:30 [10:04:30.134]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:200b319d00cfed1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:200b319d00cfed1f"}
10:04:30 [10:04:30.134]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:53883d55e77876c1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:53883d55e77876c1"}
10:04:31 [10:04:31.186]
INFO:
prover-client:proving-broker Proving Broker started
10:04:31 [10:04:31.187]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:200b319d00cfed1f settled=false
{"provingJobId":"1:BASE_PARITY:200b319d00cfed1f","status":"pending"}
10:04:31 [10:04:31.187]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:53883d55e77876c1 settled=false
{"provingJobId":"1:BASE_PARITY:53883d55e77876c1","status":"pending"}
10:04:31 [10:04:31.188]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:200b319d00cfed1f type=BASE_PARITY startedAt=1750413870134 older agent has taken job
{"provingJobId":"1:BASE_PARITY:200b319d00cfed1f"}
10:04:31 [10:04:31.189]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:200b319d00cfed1f type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:200b319d00cfed1f"}
10:04:31 [10:04:31.190]
INFO:
prover-client:proving-broker Proving Broker started
10:04:31 [10:04:31.191]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8fe07606f6c46628 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8fe07606f6c46628"}
10:04:31 [10:04:31.191]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d8318b8ad69022e epochNumber=2
{"provingJobId":"1:BASE_PARITY:4d8318b8ad69022e"}
10:04:32 [10:04:32.192]
INFO:
prover-client:proving-broker Proving Broker started
10:04:32 [10:04:32.193]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8fe07606f6c46628 settled=false
{"provingJobId":"1:BASE_PARITY:8fe07606f6c46628","status":"pending"}
10:04:32 [10:04:32.193]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4d8318b8ad69022e settled=false
{"provingJobId":"1:BASE_PARITY:4d8318b8ad69022e","status":"pending"}
10:04:32 [10:04:32.193]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8fe07606f6c46628 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:8fe07606f6c46628"}
10:04:32 [10:04:32.195]
INFO:
prover-client:proving-broker Proving Broker started
10:04:32 [10:04:32.196]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e030b19ad3f6da14 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e030b19ad3f6da14"}
10:04:32 [10:04:32.196]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81ff051c2bfd9725 epochNumber=2
{"provingJobId":"1:BASE_PARITY:81ff051c2bfd9725"}
10:04:33 [10:04:33.197]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.197]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e030b19ad3f6da14 settled=false
{"provingJobId":"1:BASE_PARITY:e030b19ad3f6da14","status":"pending"}
10:04:33 [10:04:33.198]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:81ff051c2bfd9725 settled=false
{"provingJobId":"1:BASE_PARITY:81ff051c2bfd9725","status":"pending"}
10:04:33 [10:04:33.199]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e030b19ad3f6da14 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e030b19ad3f6da14"}
10:04:33 [10:04:33.199]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e030b19ad3f6da14 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e030b19ad3f6da14"}
10:04:33 [10:04:33.201]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.201]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0282af1fba7b7c70 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0282af1fba7b7c70"}
10:04:33 [10:04:33.201]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e82420366a2e84cf epochNumber=2
{"provingJobId":"1:BASE_PARITY:e82420366a2e84cf"}
10:04:33 [10:04:33.202]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0282af1fba7b7c70 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:0282af1fba7b7c70"}
10:04:33 [10:04:33.202]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e82420366a2e84cf type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e82420366a2e84cf"}
10:04:33 [10:04:33.203]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.203]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:62c6ac6f3ecbd6c3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:62c6ac6f3ecbd6c3"}
10:04:33 [10:04:33.204]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75a4380fd8860bf1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:75a4380fd8860bf1"}
10:04:33 [10:04:33.204]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:62c6ac6f3ecbd6c3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:62c6ac6f3ecbd6c3"}
10:04:33 [10:04:33.204]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:62c6ac6f3ecbd6c3 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:62c6ac6f3ecbd6c3"}
10:04:33 [10:04:33.204]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:75a4380fd8860bf1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:75a4380fd8860bf1"}
10:04:33 [10:04:33.205]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.206]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4256dd5ec70d1f67 err=test error
{"provingJoId":"1:BASE_PARITY:4256dd5ec70d1f67"}
10:04:33 [10:04:33.206]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.207]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7d3df61e635ad92c not found
{"provingJobId":"1:BASE_PARITY:7d3df61e635ad92c"}
10:04:33 [10:04:33.208]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.209]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f157bc739feeca9e epochNumber=1
{"provingJobId":"1:BASE_PARITY:f157bc739feeca9e"}
10:04:33 [10:04:33.210]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.210]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:98a26a3217ae493d epochNumber=1
{"provingJobId":"1:BASE_PARITY:98a26a3217ae493d"}
10:04:33 [10:04:33.312]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:98a26a3217ae493d timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:98a26a3217ae493d"}
10:04:33 [10:04:33.320]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.321]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a9a030e2a8b331a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a9a030e2a8b331a5"}
10:04:33 [10:04:33.423]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a9a030e2a8b331a5 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a9a030e2a8b331a5"}
10:04:33 [10:04:33.430]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8375d88a2625a37e epochNumber=10
{"provingJobId":"1:BASE_PARITY:8375d88a2625a37e"}
10:04:33 [10:04:33.450]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:33 [10:04:33.557]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.558]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:89f6c37e054bd847 epochNumber=1
{"provingJobId":"1:BASE_PARITY:89f6c37e054bd847"}
10:04:33 [10:04:33.712]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:89f6c37e054bd847 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:89f6c37e054bd847"}
10:04:33 [10:04:33.762]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50f068aa49e1679e epochNumber=1
{"provingJobId":"1:BASE_PARITY:50f068aa49e1679e"}
10:04:33 [10:04:33.763]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:50f068aa49e1679e type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:50f068aa49e1679e"}
10:04:33 [10:04:33.765]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.766]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2a59f90bca82b98 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f2a59f90bca82b98"}
10:04:33 [10:04:33.767]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f2a59f90bca82b98 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f2a59f90bca82b98"}
10:04:33 [10:04:33.768]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f2a59f90bca82b98 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:f2a59f90bca82b98"}
10:04:33 [10:04:33.771]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:68a2909c963829fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:68a2909c963829fd"}
10:04:33 [10:04:33.773]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:68a2909c963829fd type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:68a2909c963829fd"}
10:04:33 [10:04:33.774]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.775]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:682ef7eb300ea3d6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:682ef7eb300ea3d6"}
10:04:33 [10:04:33.776]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:682ef7eb300ea3d6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:682ef7eb300ea3d6"}
10:04:33 [10:04:33.776]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7e7180addda0309 epochNumber=3
{"provingJobId":"1:BASE_PARITY:d7e7180addda0309"}
10:04:33 [10:04:33.800]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:33 [10:04:33.802]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:682ef7eb300ea3d6 err=test error
{"provingJoId":"1:BASE_PARITY:682ef7eb300ea3d6"}
10:04:33 [10:04:33.803]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.804]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b8f42dfd0d0a0934 settled=false
{"provingJobId":"1:BASE_PARITY:b8f42dfd0d0a0934","status":"pending"}
10:04:33 [10:04:33.804]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dec38c0e02c10f83 settled=false
{"provingJobId":"1:BASE_PARITY:dec38c0e02c10f83","status":"pending"}
10:04:33 [10:04:33.807]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.808]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9293a98eea17915d settled=true
{"provingJobId":"1:BASE_PARITY:9293a98eea17915d","status":"fulfilled"}
10:04:33 [10:04:33.808]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:a047579a1474e6bc settled=true
{"provingJobId":"2:BASE_PARITY:a047579a1474e6bc","status":"fulfilled"}
10:04:33 [10:04:33.810]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.810]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1ff3c68027440c08 settled=true
{"provingJobId":"1:BASE_PARITY:1ff3c68027440c08","status":"fulfilled"}
10:04:33 [10:04:33.811]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9daf0ff086acb1bb settled=false
{"provingJobId":"1:BASE_PARITY:9daf0ff086acb1bb","status":"pending"}
10:04:33 [10:04:33.812]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.814]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e4a66fb953963ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e4a66fb953963ea"}
10:04:33 [10:04:33.816]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.817]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef9934ad3eb34f61 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef9934ad3eb34f61"}
10:04:33 [10:04:33.818]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:ef9934ad3eb34f61: Error: db error: Error: db error
10:04:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:ef9934ad3eb34f61"}
10:04:33 [10:04:33.820]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.820]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c335745f5c5a87d epochNumber=1
{"provingJobId":"1:BASE_PARITY:6c335745f5c5a87d"}
10:04:33 [10:04:33.821]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6c335745f5c5a87d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6c335745f5c5a87d"}
10:04:33 [10:04:33.821]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6c335745f5c5a87d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:6c335745f5c5a87d"}
10:04:33 [10:04:33.822]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.823]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3ec42e21cb8bad45 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3ec42e21cb8bad45"}
10:04:33 [10:04:33.823]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3ec42e21cb8bad45 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3ec42e21cb8bad45"}
10:04:33 [10:04:33.823]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3ec42e21cb8bad45 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3ec42e21cb8bad45"}
10:04:33 [10:04:33.824]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:3ec42e21cb8bad45: Error: db error
10:04:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:3ec42e21cb8bad45"}
10:04:33 [10:04:33.826]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.826]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ce3b1e8bf9e1bf08 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ce3b1e8bf9e1bf08"}
10:04:33 [10:04:33.827]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ce3b1e8bf9e1bf08 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ce3b1e8bf9e1bf08"}
10:04:33 [10:04:33.827]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ce3b1e8bf9e1bf08 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:ce3b1e8bf9e1bf08"}
10:04:33 [10:04:33.828]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.829]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:013780ce72e44fdb epochNumber=1
{"provingJobId":"1:BASE_PARITY:013780ce72e44fdb"}
10:04:33 [10:04:33.829]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:013780ce72e44fdb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:013780ce72e44fdb"}
10:04:33 [10:04:33.829]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:013780ce72e44fdb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:013780ce72e44fdb"}
10:04:33 [10:04:33.830]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:013780ce72e44fdb jobErr=test error: Error: db error
10:04:33 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:013780ce72e44fdb"}
10:04:33 [10:04:33.832]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.832]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f9bc101866378f20 not found
{"provingJobId":"1:BASE_PARITY:f9bc101866378f20"}
10:04:33 [10:04:33.834]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.834]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7ef3b3dc1ba87a5d err=test error
{"provingJoId":"1:BASE_PARITY:7ef3b3dc1ba87a5d"}
10:04:33 [10:04:33.835]
INFO:
prover-client:proving-broker Proving Broker started
10:04:33 [10:04:33.846]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:730f7eac44be272a epochNumber=1
{"provingJobId":"1:BASE_PARITY:730f7eac44be272a"}
10:04:33 [10:04:33.846]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:730f7eac44be272a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:730f7eac44be272a"}
10:04:33 [10:04:33.846]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:730f7eac44be272a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:730f7eac44be272a"}
10:04:33 [10:04:33.857]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:29a527b3ade9d8ac epochNumber=2
{"provingJobId":"2:BASE_PARITY:29a527b3ade9d8ac"}
10:04:33 [10:04:33.858]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:29a527b3ade9d8ac type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:29a527b3ade9d8ac"}
10:04:33 [10:04:33.859]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:29a527b3ade9d8ac type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:29a527b3ade9d8ac"}
10:04:33 [10:04:33.922]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:d664a14d28c10117 epochNumber=3
{"provingJobId":"3:BASE_PARITY:d664a14d28c10117"}
10:04:33 [10:04:33.939]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:33 [10:04:33.986]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:62bf6eaf2b3af829 epochNumber=4
{"provingJobId":"4:BASE_PARITY:62bf6eaf2b3af829"}
10:04:33 [10:04:33.989]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:34 [10:04:34.048]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:2a4f6d0d7ebaed20 epochNumber=5
{"provingJobId":"5:BASE_PARITY:2a4f6d0d7ebaed20"}
10:04:34 [10:04:34.065]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:34 [10:04:34.103]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.104]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:343b8482d5aa2b83 epochNumber=42
{"provingJobId":"1:BASE_PARITY:343b8482d5aa2b83"}
10:04:34 [10:04:34.105]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testF1xtr0/42 with map size 1073741824
10:04:34 [10:04:34.109]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.141]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:1d70057a25638292 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1d70057a25638292"}
10:04:34 [10:04:34.150]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.151]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1cd670004ca0aca3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1cd670004ca0aca3"}
10:04:34 [10:04:34.153]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testP1tTjS/1 with map size 1073741824
10:04:34 [10:04:34.156]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.179]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83cab83f3eab26f6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:83cab83f3eab26f6"}
10:04:34 [10:04:34.189]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.190]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3eee9b8252c27db7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3eee9b8252c27db7"}
10:04:34 [10:04:34.192]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWo1xBr/1 with map size 1073741824
10:04:34 [10:04:34.196]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.217]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3eee9b8252c27db7 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3eee9b8252c27db7"}
10:04:34 [10:04:34.221]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.222]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2e1ec183b7cf0207 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2e1ec183b7cf0207"}
10:04:34 [10:04:34.225]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testR0U0Hv/1 with map size 1073741824
10:04:34 [10:04:34.229]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.266]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2e1ec183b7cf0207 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2e1ec183b7cf0207"}
10:04:34 [10:04:34.267]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2e1ec183b7cf0207 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2e1ec183b7cf0207"}
10:04:34 [10:04:34.273]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2e1ec183b7cf0207 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:2e1ec183b7cf0207"}
10:04:34 [10:04:34.283]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.284]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e1820e0591a065dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:e1820e0591a065dc"}
10:04:34 [10:04:34.284]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqgzliX/1 with map size 1073741824
10:04:34 [10:04:34.286]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.308]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.311]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.312]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c8320fd415acabf epochNumber=1
{"provingJobId":"1:BASE_PARITY:1c8320fd415acabf"}
10:04:34 [10:04:34.313]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5naZl8/1 with map size 1073741824
10:04:34 [10:04:34.315]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.333]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1c8320fd415acabf
{"provingJobId":"1:BASE_PARITY:1c8320fd415acabf"}
10:04:34 [10:04:34.333]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1c8320fd415acabf type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1c8320fd415acabf"}
10:04:34 [10:04:34.334]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1c8320fd415acabf type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:1c8320fd415acabf"}
10:04:34 [10:04:34.345]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.346]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df083f01f21b9f41 epochNumber=1
{"provingJobId":"1:BASE_PARITY:df083f01f21b9f41"}
10:04:34 [10:04:34.349]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9mFqpd/1 with map size 1073741824
10:04:34 [10:04:34.356]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.376]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:df083f01f21b9f41
{"provingJobId":"1:BASE_PARITY:df083f01f21b9f41"}
10:04:34 [10:04:34.377]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:df083f01f21b9f41 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:df083f01f21b9f41"}
10:04:34 [10:04:34.388]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.390]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b1979cecce361ef1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b1979cecce361ef1"}
10:04:34 [10:04:34.391]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGkc8C1/1 with map size 1073741824
10:04:34 [10:04:34.393]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.411]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b1979cecce361ef1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b1979cecce361ef1"}
10:04:34 [10:04:34.412]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b1979cecce361ef1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b1979cecce361ef1"}
10:04:34 [10:04:34.422]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.423]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6822aff6ad8c9213 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6822aff6ad8c9213"}
10:04:34 [10:04:34.424]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYnz1mP/1 with map size 1073741824
10:04:34 [10:04:34.427]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.451]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6822aff6ad8c9213 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6822aff6ad8c9213"}
10:04:34 [10:04:34.452]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6822aff6ad8c9213 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6822aff6ad8c9213"}
10:04:34 [10:04:34.463]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.464]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2fcd80e0032c98f epochNumber=0
{"provingJobId":"1:BASE_PARITY:e2fcd80e0032c98f"}
10:04:34 [10:04:34.464]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testcQORSR/0 with map size 1073741824
10:04:34 [10:04:34.466]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.482]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e2fcd80e0032c98f epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e2fcd80e0032c98f"}
10:04:34 [10:04:34.483]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:999a689b9db2c34b epochNumber=0
{"provingJobId":"1:BASE_PARITY:999a689b9db2c34b"}
10:04:34 [10:04:34.483]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e2fcd80e0032c98f epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e2fcd80e0032c98f"}
10:04:34 [10:04:34.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8da8ceb16f35cc30 epochNumber=0
{"provingJobId":"1:BASE_PARITY:8da8ceb16f35cc30"}
10:04:34 [10:04:34.497]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.499]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.500]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ceb419392c9b09fb epochNumber=2
{"provingJobId":"1:BASE_PARITY:ceb419392c9b09fb"}
10:04:34 [10:04:34.500]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtHdyBN/2 with map size 1073741824
10:04:34 [10:04:34.502]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.517]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0596515baaa53334 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0596515baaa53334"}
10:04:34 [10:04:34.518]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtHdyBN/1 with map size 1073741824
10:04:34 [10:04:34.521]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.544]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fb8fa47c4eae8c78 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fb8fa47c4eae8c78"}
10:04:34 [10:04:34.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoFN3rD/1 with map size 1073741824
10:04:34 [10:04:34.546]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.564]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.565]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4737be11f1d269e epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4737be11f1d269e"}
10:04:34 [10:04:34.566]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-tests4kXul/1 with map size 1073741824
10:04:34 [10:04:34.567]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.583]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:399657dfbb429223 epochNumber=1
{"provingJobId":"1:BASE_PARITY:399657dfbb429223"}
10:04:34 [10:04:34.589]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c1fb450890f46bd epochNumber=2
{"provingJobId":"1:BASE_PARITY:4c1fb450890f46bd"}
10:04:34 [10:04:34.590]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-tests4kXul/2 with map size 1073741824
10:04:34 [10:04:34.592]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.610]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6b8b2ffe2826abbf epochNumber=1
{"provingJobId":"1:BASE_PARITY:6b8b2ffe2826abbf"}
10:04:34 [10:04:34.621]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.622]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4fedb4402e31946e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4fedb4402e31946e"}
10:04:34 [10:04:34.623]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeErntO/1 with map size 1073741824
10:04:34 [10:04:34.626]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.648]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:70641aa4c7b61c2a epochNumber=1
{"provingJobId":"1:BASE_PARITY:70641aa4c7b61c2a"}
10:04:34 [10:04:34.656]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:edddf1c42025abeb epochNumber=2
{"provingJobId":"1:BASE_PARITY:edddf1c42025abeb"}
10:04:34 [10:04:34.657]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testeErntO/2 with map size 1073741824
10:04:34 [10:04:34.660]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.676]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aade3a51c19f8ef1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aade3a51c19f8ef1"}
10:04:34 [10:04:34.685]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.686]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dd9f3893f496b7d7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dd9f3893f496b7d7"}
10:04:34 [10:04:34.686]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPf99UJ/1 with map size 1073741824
10:04:34 [10:04:34.689]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.709]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3896863ea5c56bd epochNumber=1
{"provingJobId":"1:BASE_PARITY:f3896863ea5c56bd"}
10:04:34 [10:04:34.716]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77d099ed0d3baa31 epochNumber=2
{"provingJobId":"1:BASE_PARITY:77d099ed0d3baa31"}
10:04:34 [10:04:34.718]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testPf99UJ/2 with map size 1073741824
10:04:34 [10:04:34.721]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.749]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be68160d64eac22e epochNumber=1
{"provingJobId":"1:BASE_PARITY:be68160d64eac22e"}
10:04:34 [10:04:34.761]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.762]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13c013836d87374e epochNumber=1
{"provingJobId":"1:BASE_PARITY:13c013836d87374e"}
10:04:34 [10:04:34.763]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOy3OAI/1 with map size 1073741824
10:04:34 [10:04:34.765]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09819c1602e8d250 epochNumber=1
{"provingJobId":"1:BASE_PARITY:09819c1602e8d250"}
10:04:34 [10:04:34.788]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:13c013836d87374e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:13c013836d87374e"}
10:04:34 [10:04:34.798]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.799]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4de756e141460734 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4de756e141460734"}
10:04:34 [10:04:34.800]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testINsrC2/1 with map size 1073741824
10:04:34 [10:04:34.802]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.821]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f92fcc889e899093 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f92fcc889e899093"}
10:04:34 [10:04:34.828]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4de756e141460734 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:4de756e141460734"}
10:04:34 [10:04:34.843]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.844]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2006bf39eab7714 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e2006bf39eab7714"}
10:04:34 [10:04:34.849]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvo8GP3/1 with map size 1073741824
10:04:34 [10:04:34.851]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.869]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:83c523db5aa2149f epochNumber=1
{"provingJobId":"1:BASE_PARITY:83c523db5aa2149f"}
10:04:34 [10:04:34.876]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e2006bf39eab7714 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:e2006bf39eab7714"}
10:04:34 [10:04:34.880]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.881]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60567af31f1f2f0a epochNumber=1
{"provingJobId":"1:BASE_PARITY:60567af31f1f2f0a"}
10:04:34 [10:04:34.882]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3kMzy3/1 with map size 1073741824
10:04:34 [10:04:34.884]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.901]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:60567af31f1f2f0a
{"provingJobId":"1:BASE_PARITY:60567af31f1f2f0a"}
10:04:34 [10:04:34.901]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:60567af31f1f2f0a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:60567af31f1f2f0a"}
10:04:34 [10:04:34.907]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:44c913d6e958d558 epochNumber=1
{"provingJobId":"1:BASE_PARITY:44c913d6e958d558"}
10:04:34 [10:04:34.913]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:60567af31f1f2f0a has already been completed
{"provingJobId":"1:BASE_PARITY:60567af31f1f2f0a"}
10:04:34 [10:04:34.919]
INFO:
prover-client:proving-broker Proving Broker started
10:04:34 [10:04:34.920]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eeedf678145dbc7b epochNumber=1
{"provingJobId":"1:BASE_PARITY:eeedf678145dbc7b"}
10:04:34 [10:04:34.920]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testT741rh/1 with map size 1073741824
10:04:34 [10:04:34.922]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:34 [10:04:34.945]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b4e5aac1989f3f34 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b4e5aac1989f3f34"}
10:04:34 [10:04:34.946]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testT741rh/2 with map size 1073741824
10:04:34 [10:04:34.948]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:36 [10:04:36.018]
INFO:
prover-client:proving-broker Proving Broker started
10:04:36 [10:04:36.023]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:eeedf678145dbc7b settled=false
{"provingJobId":"1:BASE_PARITY:eeedf678145dbc7b","status":"pending"}
10:04:36 [10:04:36.024]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b4e5aac1989f3f34 settled=false
{"provingJobId":"1:BASE_PARITY:b4e5aac1989f3f34","status":"pending"}
10:04:36 [10:04:36.025]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eeedf678145dbc7b type=BASE_PARITY startedAt=1750413874967 older agent has taken job
{"provingJobId":"1:BASE_PARITY:eeedf678145dbc7b"}
10:04:36 [10:04:36.025]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:eeedf678145dbc7b type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:eeedf678145dbc7b"}
10:04:36 [10:04:36.028]
INFO:
prover-client:proving-broker Proving Broker started
10:04:36 [10:04:36.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3eb006e03740151 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3eb006e03740151"}
10:04:36 [10:04:36.031]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4RpWtK/1 with map size 1073741824
10:04:36 [10:04:36.037]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:36 [10:04:36.063]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c22b30c1ebcfd1da epochNumber=2
{"provingJobId":"1:BASE_PARITY:c22b30c1ebcfd1da"}
10:04:36 [10:04:36.064]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4RpWtK/2 with map size 1073741824
10:04:36 [10:04:36.066]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:37 [10:04:37.085]
INFO:
prover-client:proving-broker Proving Broker started
10:04:37 [10:04:37.087]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d3eb006e03740151 settled=false
{"provingJobId":"1:BASE_PARITY:d3eb006e03740151","status":"pending"}
10:04:37 [10:04:37.091]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c22b30c1ebcfd1da settled=false
{"provingJobId":"1:BASE_PARITY:c22b30c1ebcfd1da","status":"pending"}
10:04:37 [10:04:37.092]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d3eb006e03740151 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:d3eb006e03740151"}
10:04:37 [10:04:37.096]
INFO:
prover-client:proving-broker Proving Broker started
10:04:37 [10:04:37.097]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d51654ed9ae6161 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d51654ed9ae6161"}
10:04:37 [10:04:37.097]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test46w2uZ/1 with map size 1073741824
10:04:37 [10:04:37.100]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:37 [10:04:37.119]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9b68a2d1e84338c epochNumber=2
{"provingJobId":"1:BASE_PARITY:b9b68a2d1e84338c"}
10:04:37 [10:04:37.120]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test46w2uZ/2 with map size 1073741824
10:04:37 [10:04:37.126]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.151]
INFO:
prover-client:proving-broker Proving Broker started
10:04:38 [10:04:38.154]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8d51654ed9ae6161 settled=false
{"provingJobId":"1:BASE_PARITY:8d51654ed9ae6161","status":"pending"}
10:04:38 [10:04:38.155]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b9b68a2d1e84338c settled=false
{"provingJobId":"1:BASE_PARITY:b9b68a2d1e84338c","status":"pending"}
10:04:38 [10:04:38.157]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8d51654ed9ae6161 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8d51654ed9ae6161"}
10:04:38 [10:04:38.158]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8d51654ed9ae6161 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8d51654ed9ae6161"}
10:04:38 [10:04:38.171]
INFO:
prover-client:proving-broker Proving Broker started
10:04:38 [10:04:38.172]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cd4a7f5f635d4d2f epochNumber=1
{"provingJobId":"1:BASE_PARITY:cd4a7f5f635d4d2f"}
10:04:38 [10:04:38.172]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3yICGe/1 with map size 1073741824
10:04:38 [10:04:38.175]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.193]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:408f33bb0f3670d3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:408f33bb0f3670d3"}
10:04:38 [10:04:38.194]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test3yICGe/2 with map size 1073741824
10:04:38 [10:04:38.198]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.218]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cd4a7f5f635d4d2f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:cd4a7f5f635d4d2f"}
10:04:38 [10:04:38.229]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:408f33bb0f3670d3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:408f33bb0f3670d3"}
10:04:38 [10:04:38.248]
INFO:
prover-client:proving-broker Proving Broker started
10:04:38 [10:04:38.249]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df0c6155cde828bc epochNumber=1
{"provingJobId":"1:BASE_PARITY:df0c6155cde828bc"}
10:04:38 [10:04:38.250]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnGzVCx/1 with map size 1073741824
10:04:38 [10:04:38.255]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.272]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7a003f84bbbdd33 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e7a003f84bbbdd33"}
10:04:38 [10:04:38.273]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnGzVCx/2 with map size 1073741824
10:04:38 [10:04:38.277]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.297]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:df0c6155cde828bc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:df0c6155cde828bc"}
10:04:38 [10:04:38.298]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:df0c6155cde828bc type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:df0c6155cde828bc"}
10:04:38 [10:04:38.304]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e7a003f84bbbdd33 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e7a003f84bbbdd33"}
10:04:38 [10:04:38.314]
INFO:
prover-client:proving-broker Proving Broker started
10:04:38 [10:04:38.315]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e2f717bbf53023ed err=test error
{"provingJoId":"1:BASE_PARITY:e2f717bbf53023ed"}
10:04:38 [10:04:38.319]
INFO:
prover-client:proving-broker Proving Broker started
10:04:38 [10:04:38.320]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:56ea2546400facfc not found
{"provingJobId":"1:BASE_PARITY:56ea2546400facfc"}
10:04:38 [10:04:38.322]
INFO:
prover-client:proving-broker Proving Broker started
10:04:38 [10:04:38.323]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:072b6a35fb265f3a epochNumber=1
{"provingJobId":"1:BASE_PARITY:072b6a35fb265f3a"}
10:04:38 [10:04:38.324]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxoQkNT/1 with map size 1073741824
10:04:38 [10:04:38.326]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.354]
INFO:
prover-client:proving-broker Proving Broker started
10:04:38 [10:04:38.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8b7cecf2118d4b0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:8b7cecf2118d4b0b"}
10:04:38 [10:04:38.359]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testajO2Kp/1 with map size 1073741824
10:04:38 [10:04:38.362]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.486]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8b7cecf2118d4b0b timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:8b7cecf2118d4b0b"}
10:04:38 [10:04:38.495]
INFO:
prover-client:proving-broker Proving Broker started
10:04:38 [10:04:38.496]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:188b88b27b555991 epochNumber=1
{"provingJobId":"1:BASE_PARITY:188b88b27b555991"}
10:04:38 [10:04:38.499]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjYLdzD/1 with map size 1073741824
10:04:38 [10:04:38.502]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.622]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:188b88b27b555991 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:188b88b27b555991"}
10:04:38 [10:04:38.629]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f3ea4e928abff7af epochNumber=10
{"provingJobId":"1:BASE_PARITY:f3ea4e928abff7af"}
10:04:38 [10:04:38.630]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testjYLdzD/10 with map size 1073741824
10:04:38 [10:04:38.636]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.673]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:38 [10:04:38.674]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:04:38 [10:04:38.782]
INFO:
prover-client:proving-broker Proving Broker started
10:04:38 [10:04:38.783]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ea607bfe9354a132 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ea607bfe9354a132"}
10:04:38 [10:04:38.783]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3bfQcS/1 with map size 1073741824
10:04:38 [10:04:38.785]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:38 [10:04:38.963]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ea607bfe9354a132 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:ea607bfe9354a132"}
10:04:39 [10:04:39.013]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.014]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:75a634c6626b8a8c epochNumber=1
{"provingJobId":"1:BASE_PARITY:75a634c6626b8a8c"}
10:04:39 [10:04:39.017]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkEyvJC/1 with map size 1073741824
10:04:39 [10:04:39.020]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.040]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:75a634c6626b8a8c type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:75a634c6626b8a8c"}
10:04:39 [10:04:39.046]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.047]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ab3fa9ed2f10c272 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ab3fa9ed2f10c272"}
10:04:39 [10:04:39.048]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJnbLaN/1 with map size 1073741824
10:04:39 [10:04:39.050]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.072]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ab3fa9ed2f10c272 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:ab3fa9ed2f10c272"}
10:04:39 [10:04:39.073]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ab3fa9ed2f10c272 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:ab3fa9ed2f10c272"}
10:04:39 [10:04:39.083]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.084]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:88d9babe538849cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:88d9babe538849cb"}
10:04:39 [10:04:39.085]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDW9BBY/1 with map size 1073741824
10:04:39 [10:04:39.087]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.104]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:88d9babe538849cb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:88d9babe538849cb"}
10:04:39 [10:04:39.115]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.116]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2b755afd486cf1b9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2b755afd486cf1b9"}
10:04:39 [10:04:39.117]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBtSMPm/1 with map size 1073741824
10:04:39 [10:04:39.119]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.149]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:2b755afd486cf1b9 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:2b755afd486cf1b9"}
10:04:39 [10:04:39.151]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71b3154787a98fb0 epochNumber=3
{"provingJobId":"1:BASE_PARITY:71b3154787a98fb0"}
10:04:39 [10:04:39.153]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testBtSMPm/3 with map size 1073741824
10:04:39 [10:04:39.157]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.191]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:39 [10:04:39.191]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:04:39 [10:04:39.202]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:2b755afd486cf1b9 err=test error
{"provingJoId":"1:BASE_PARITY:2b755afd486cf1b9"}
10:04:39 [10:04:39.207]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQ7v4zi/1 with map size 1073741824
10:04:39 [10:04:39.210]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.231]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQ7v4zi/2 with map size 1073741824
10:04:39 [10:04:39.240]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.259]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.261]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:018f8e39b923d840 settled=false
{"provingJobId":"1:BASE_PARITY:018f8e39b923d840","status":"pending"}
10:04:39 [10:04:39.263]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4d8d9478949e5571 settled=false
{"provingJobId":"1:BASE_PARITY:4d8d9478949e5571","status":"pending"}
10:04:39 [10:04:39.271]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1EepHq/1 with map size 1073741824
10:04:39 [10:04:39.275]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.296]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test1EepHq/2 with map size 1073741824
10:04:39 [10:04:39.299]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.329]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.332]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4b69be69155ab7c3 settled=true
{"provingJobId":"1:BASE_PARITY:4b69be69155ab7c3","status":"fulfilled"}
10:04:39 [10:04:39.334]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:2209e308ab9bb49d settled=true
{"provingJobId":"2:BASE_PARITY:2209e308ab9bb49d","status":"fulfilled"}
10:04:39 [10:04:39.350]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0x9G9Q/1 with map size 1073741824
10:04:39 [10:04:39.356]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.380]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0x9G9Q/2 with map size 1073741824
10:04:39 [10:04:39.384]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.405]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.409]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0fcb08ce727e85d4 settled=true
{"provingJobId":"1:BASE_PARITY:0fcb08ce727e85d4","status":"fulfilled"}
10:04:39 [10:04:39.412]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7b95694dcbbb4f3f settled=false
{"provingJobId":"1:BASE_PARITY:7b95694dcbbb4f3f","status":"pending"}
10:04:39 [10:04:39.417]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.418]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b80a5f65ffd6cab epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b80a5f65ffd6cab"}
10:04:39 [10:04:39.419]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5zyjPi/1 with map size 1073741824
10:04:39 [10:04:39.421]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.445]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.445]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bf6b6c46071d30fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:bf6b6c46071d30fe"}
10:04:39 [10:04:39.446]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:bf6b6c46071d30fe: Error: db error: Error: db error
10:04:39 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:bf6b6c46071d30fe"}
10:04:39 [10:04:39.451]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.451]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:99c58ec5a3251068 epochNumber=1
{"provingJobId":"1:BASE_PARITY:99c58ec5a3251068"}
10:04:39 [10:04:39.453]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test28OhJI/1 with map size 1073741824
10:04:39 [10:04:39.458]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.473]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:99c58ec5a3251068 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:99c58ec5a3251068"}
10:04:39 [10:04:39.474]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:99c58ec5a3251068 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:99c58ec5a3251068"}
10:04:39 [10:04:39.483]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.484]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5de6febffab4a1ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:5de6febffab4a1ca"}
10:04:39 [10:04:39.484]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9ifELR/1 with map size 1073741824
10:04:39 [10:04:39.487]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.503]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5de6febffab4a1ca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5de6febffab4a1ca"}
10:04:39 [10:04:39.504]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5de6febffab4a1ca type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5de6febffab4a1ca"}
10:04:39 [10:04:39.505]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:5de6febffab4a1ca: Error: db error
10:04:39 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:5de6febffab4a1ca"}
10:04:39 [10:04:39.509]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.510]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a9166523d549157 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3a9166523d549157"}
10:04:39 [10:04:39.511]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3sqAsw/1 with map size 1073741824
10:04:39 [10:04:39.514]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.534]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3a9166523d549157 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3a9166523d549157"}
10:04:39 [10:04:39.535]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3a9166523d549157 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3a9166523d549157"}
10:04:39 [10:04:39.557]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.558]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8dfcc1904f262a58 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8dfcc1904f262a58"}
10:04:39 [10:04:39.559]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKakks1/1 with map size 1073741824
10:04:39 [10:04:39.562]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.585]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8dfcc1904f262a58 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8dfcc1904f262a58"}
10:04:39 [10:04:39.586]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8dfcc1904f262a58 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8dfcc1904f262a58"}
10:04:39 [10:04:39.587]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:8dfcc1904f262a58 jobErr=test error: Error: db error
10:04:39 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:8dfcc1904f262a58"}
10:04:39 [10:04:39.595]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.596]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3bc684cdcabc4387 not found
{"provingJobId":"1:BASE_PARITY:3bc684cdcabc4387"}
10:04:39 [10:04:39.602]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.603]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7fd8fae66c30e937 err=test error
{"provingJoId":"1:BASE_PARITY:7fd8fae66c30e937"}
10:04:39 [10:04:39.608]
INFO:
prover-client:proving-broker Proving Broker started
10:04:39 [10:04:39.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e78657712b1a50de epochNumber=1
{"provingJobId":"1:BASE_PARITY:e78657712b1a50de"}
10:04:39 [10:04:39.619]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDzwnU8/1 with map size 1073741824
10:04:39 [10:04:39.624]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.650]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e78657712b1a50de type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e78657712b1a50de"}
10:04:39 [10:04:39.651]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e78657712b1a50de type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e78657712b1a50de"}
10:04:39 [10:04:39.671]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:787a38be4a978d54 epochNumber=2
{"provingJobId":"2:BASE_PARITY:787a38be4a978d54"}
10:04:39 [10:04:39.673]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDzwnU8/2 with map size 1073741824
10:04:39 [10:04:39.675]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.699]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:787a38be4a978d54 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:787a38be4a978d54"}
10:04:39 [10:04:39.700]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:787a38be4a978d54 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:787a38be4a978d54"}
10:04:39 [10:04:39.767]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:8288e49957d6fc8b epochNumber=3
{"provingJobId":"3:BASE_PARITY:8288e49957d6fc8b"}
10:04:39 [10:04:39.769]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testDzwnU8/3 with map size 1073741824
10:04:39 [10:04:39.772]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.815]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:39 [10:04:39.815]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
10:04:39 [10:04:39.857]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:e9e36aa76f991075 epochNumber=4
{"provingJobId":"4:BASE_PARITY:e9e36aa76f991075"}
10:04:39 [10:04:39.858]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testDzwnU8/4 with map size 1073741824
10:04:39 [10:04:39.861]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.892]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:39 [10:04:39.893]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
10:04:39 [10:04:39.944]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:3b4a9936a3fb7939 epochNumber=5
{"provingJobId":"5:BASE_PARITY:3b4a9936a3fb7939"}
10:04:39 [10:04:39.946]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testDzwnU8/5 with map size 1073741824
10:04:39 [10:04:39.949]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
10:04:39 [10:04:39.971]
INFO:
prover-client:proving-broker Cleaned up jobs=1
10:04:39 [10:04:39.971]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
10:04:40
PASS src/proving_broker/proving_broker.test.ts (
13.245 s)
10:04:40 ProvingBroker
10:04:40 Producer API
10:04:40
✓ refuses stale jobs (48 ms)
10:04:40
✓ enqueues jobs (3 ms)
10:04:40
✓ ignores duplicate jobs (3 ms)
10:04:40
✓ reports correct status when enqueuing repeat jobs (3 ms)
10:04:40
✓ throws an error in case of duplicate job IDs (25 ms)
10:04:40
✓ returns not-found status for non-existing jobs (2 ms)
10:04:40
✓ cancels jobs in queue (2 ms)
10:04:40
✓ cancels jobs in-progress (2 ms)
10:04:40
✓ returns job result if successful (8 ms)
10:04:40
✓ returns job error if failed (2 ms)
10:04:40
✓ correctly returns job status for concurrent writes (2 ms)
10:04:40
✓ refuses stale jobs (46 ms)
10:04:40
✓ enqueues jobs (40 ms)
10:04:40
✓ ignores duplicate jobs (31 ms)
10:04:40
✓ reports correct status when enqueuing repeat jobs (61 ms)
10:04:40
✓ throws an error in case of duplicate job IDs (25 ms)
10:04:40
✓ returns not-found status for non-existing jobs (2 ms)
10:04:40
✓ cancels jobs in queue (34 ms)
10:04:40
✓ cancels jobs in-progress (42 ms)
10:04:40
✓ returns job result if successful (33 ms)
10:04:40
✓ returns job error if failed (41 ms)
10:04:40
✓ correctly returns job status for concurrent writes (34 ms)
10:04:40 Consumer API
10:04:40
✓ returns undefined if no jobs are available (1 ms)
10:04:40
✓ returns jobs in priority order (2 ms)
10:04:40
✓ returns undefined if no jobs are available for the given allowList (1 ms)
10:04:40
✓ returns a job if it is in the allowList (2 ms)
10:04:40
✓ returns the most important job if it is in the allowList (2 ms)
10:04:40
✓ returns any job if filter is empty (1 ms)
10:04:40
✓ returns a new job when reporting job success (1 ms)
10:04:40
✓ returns a new job when reporting permanent error (2 ms)
10:04:40
✓ returns a new job when reporting retry-able error (2 ms)
10:04:40
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
10:04:40
✓ returns a new job if job is already in progress elsewhere (1056 ms)
10:04:40
✓ avoids sending the same job to a new agent after a restart (1004 ms)
10:04:40
✓ avoids sending a completed job to a new agent after a restart (1006 ms)
10:04:40
✓ tracks job result if in progress (3 ms)
10:04:40
✓ tracks job result even if job is in queue (2 ms)
10:04:40
✓ ignores reported job error if unknown job (1 ms)
10:04:40
✓ ignores job result if unknown job (1 ms)
10:04:40
✓ returns undefined if no jobs are available (3 ms)
10:04:40
✓ returns jobs in priority order (44 ms)
10:04:40
✓ returns undefined if no jobs are available for the given allowList (21 ms)
10:04:40
✓ returns a job if it is in the allowList (56 ms)
10:04:40
✓ returns the most important job if it is in the allowList (64 ms)
10:04:40
✓ returns any job if filter is empty (74 ms)
10:04:40
✓ returns a new job when reporting job success (36 ms)
10:04:40
✓ returns a new job when reporting permanent error (44 ms)
10:04:40
✓ returns a new job when reporting retry-able error (39 ms)
10:04:40
✓ returns a new job when reporting progress if current one is cancelled (36 ms)
10:04:40
✓ returns a new job if job is already in progress elsewhere (1112 ms)
10:04:40
✓ avoids sending the same job to a new agent after a restart (1067 ms)
10:04:40
✓ avoids sending a completed job to a new agent after a restart (1073 ms)
10:04:40
✓ tracks job result if in progress (76 ms)
10:04:40
✓ tracks job result even if job is in queue (66 ms)
10:04:40
✓ ignores reported job error if unknown job (4 ms)
10:04:40
✓ ignores job result if unknown job (4 ms)
10:04:40 Timeouts
10:04:40
✓ tracks in progress jobs (1 ms)
10:04:40
✓ re-enqueues jobs that time out (109 ms)
10:04:40
✓ cancel stale jobs that time out (237 ms)
10:04:40
✓ keeps the jobs in progress while it is alive (204 ms)
10:04:40
✓ tracks in progress jobs (30 ms)
10:04:40
✓ re-enqueues jobs that time out (138 ms)
10:04:40
✓ cancel stale jobs that time out (289 ms)
10:04:40
✓ keeps the jobs in progress while it is alive (230 ms)
10:04:40 Retries
10:04:40
✓ retries jobs (4 ms)
10:04:40
✓ retries up to a maximum number of times (6 ms)
10:04:40
✓ passing retry=false does not retry (2 ms)
10:04:40
✓ does not retry if job is stale (28 ms)
10:04:40
✓ retries jobs (31 ms)
10:04:40
✓ retries up to a maximum number of times (39 ms)
10:04:40
✓ passing retry=false does not retry (32 ms)
10:04:40
✓ does not retry if job is stale (91 ms)
10:04:40 Database management
10:04:40
✓ re-enqueues proof requests on start (3 ms)
10:04:40
✓ restores proof results on start (3 ms)
10:04:40
✓ only re-enqueues unfinished jobs (3 ms)
10:04:40
✓ saves job when enqueued (3 ms)
10:04:40
✓ does not retain job if database fails to save (3 ms)
10:04:40
✓ saves job result (3 ms)
10:04:40
✓ saves result even if database fails to save (3 ms)
10:04:40
✓ saves job error (3 ms)
10:04:40
✓ saves job error even if database fails to save (3 ms)
10:04:40
✓ does not save job result if job is unknown (2 ms)
10:04:40
✓ does not save job error if job is unknown (2 ms)
10:04:40
✓ cleans up old jobs periodically (265 ms)
10:04:40
✓ re-enqueues proof requests on start (62 ms)
10:04:40
✓ restores proof results on start (74 ms)
10:04:40
✓ only re-enqueues unfinished jobs (73 ms)
10:04:40
✓ saves job when enqueued (27 ms)
10:04:40
✓ does not retain job if database fails to save (4 ms)
10:04:40
✓ saves job result (33 ms)
10:04:40
✓ saves result even if database fails to save (25 ms)
10:04:40
✓ saves job error (38 ms)
10:04:40
✓ saves job error even if database fails to save (44 ms)
10:04:40
✓ does not save job result if job is unknown (7 ms)
10:04:40
✓ does not save job error if job is unknown (6 ms)
10:04:40
✓ cleans up old jobs periodically (416 ms)
10:04:40
10:04:40
Test Suites: 1 passed, 1 total
10:04:40
Tests: 96 passed, 96 total
10:04:40
Snapshots: 0 total
10:04:40
Time: 13.335 s
10:04:40
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
10:04:40
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?