Parent Log:
http://ci.aztec-labs.com/00d5d0cbbd477ac8
Command: dd02bf67482af96b yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/12b762635a976e343151861f393574b7b5ce7d52
Env: REF_NAME=gh-readonly-queue/next/pr-15319-e340440b20c8e6058b811ce607aaf8447cf11c15 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Thu Jun 26 19:02:00 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15319_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
19:02:04 [19:02:04.864]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.872]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bc24f8fa40a2db5d epochNumber=42
{"provingJobId":"1:BASE_PARITY:bc24f8fa40a2db5d"}
19:02:04 [19:02:04.877]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:7d8b3c34d5328397 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7d8b3c34d5328397"}
19:02:04 [19:02:04.912]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.913]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d6ffd1ae2a30125a epochNumber=1
{"provingJobId":"1:BASE_PARITY:d6ffd1ae2a30125a"}
19:02:04 [19:02:04.914]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:baa20441203bfb77 epochNumber=1
{"provingJobId":"1:BASE_PARITY:baa20441203bfb77"}
19:02:04 [19:02:04.916]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.917]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e2b3c2b122d85cf4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e2b3c2b122d85cf4"}
19:02:04 [19:02:04.918]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e2b3c2b122d85cf4 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:e2b3c2b122d85cf4"}
19:02:04 [19:02:04.920]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.921]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71ea908f03897fdf epochNumber=1
{"provingJobId":"1:BASE_PARITY:71ea908f03897fdf"}
19:02:04 [19:02:04.922]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:71ea908f03897fdf epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:71ea908f03897fdf"}
19:02:04 [19:02:04.923]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:71ea908f03897fdf type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:71ea908f03897fdf"}
19:02:04 [19:02:04.924]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:71ea908f03897fdf epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:71ea908f03897fdf"}
19:02:04 [19:02:04.925]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.926]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eada4c5c2a1cef5f epochNumber=1
{"provingJobId":"1:BASE_PARITY:eada4c5c2a1cef5f"}
19:02:04 [19:02:04.955]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.958]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.959]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d8f738b9d12f8eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:2d8f738b9d12f8eb"}
19:02:04 [19:02:04.960]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2d8f738b9d12f8eb
{"provingJobId":"1:BASE_PARITY:2d8f738b9d12f8eb"}
19:02:04 [19:02:04.961]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2d8f738b9d12f8eb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2d8f738b9d12f8eb"}
19:02:04 [19:02:04.961]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2d8f738b9d12f8eb type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2d8f738b9d12f8eb"}
19:02:04 [19:02:04.963]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.964]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:90caddcc72406a80 epochNumber=1
{"provingJobId":"1:BASE_PARITY:90caddcc72406a80"}
19:02:04 [19:02:04.964]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:90caddcc72406a80
{"provingJobId":"1:BASE_PARITY:90caddcc72406a80"}
19:02:04 [19:02:04.965]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:90caddcc72406a80 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:90caddcc72406a80"}
19:02:04 [19:02:04.977]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.978]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:daed55854dd0dc43 epochNumber=1
{"provingJobId":"1:BASE_PARITY:daed55854dd0dc43"}
19:02:04 [19:02:04.979]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:daed55854dd0dc43 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:daed55854dd0dc43"}
19:02:04 [19:02:04.979]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:daed55854dd0dc43 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:daed55854dd0dc43"}
19:02:04 [19:02:04.981]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.981]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f0c0c9b3fdbc518 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5f0c0c9b3fdbc518"}
19:02:04 [19:02:04.982]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5f0c0c9b3fdbc518 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5f0c0c9b3fdbc518"}
19:02:04 [19:02:04.982]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5f0c0c9b3fdbc518 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:5f0c0c9b3fdbc518"}
19:02:04 [19:02:04.984]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.984]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5dcf8b311c83e8f1 epochNumber=0
{"provingJobId":"1:BASE_PARITY:5dcf8b311c83e8f1"}
19:02:04 [19:02:04.984]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5dcf8b311c83e8f1 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5dcf8b311c83e8f1"}
19:02:04 [19:02:04.985]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c35abedffa284f9c epochNumber=0
{"provingJobId":"1:BASE_PARITY:c35abedffa284f9c"}
19:02:04 [19:02:04.985]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:5dcf8b311c83e8f1 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:5dcf8b311c83e8f1"}
19:02:04 [19:02:04.985]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be4362b49c75f436 epochNumber=0
{"provingJobId":"1:BASE_PARITY:be4362b49c75f436"}
19:02:04 [19:02:04.987]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.988]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.989]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7ea590111c57f8b epochNumber=2
{"provingJobId":"1:BASE_PARITY:e7ea590111c57f8b"}
19:02:04 [19:02:04.989]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:caf3cbfcc01a5586 epochNumber=1
{"provingJobId":"1:BASE_PARITY:caf3cbfcc01a5586"}
19:02:04 [19:02:04.991]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.991]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cbdd097a29c4b9cf epochNumber=1
{"provingJobId":"1:BASE_PARITY:cbdd097a29c4b9cf"}
19:02:04 [19:02:04.992]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.993]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f6d2865d78236c87 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f6d2865d78236c87"}
19:02:04 [19:02:04.993]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87f183c467f74fa0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87f183c467f74fa0"}
19:02:04 [19:02:04.993]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e61e5432c1fde3f epochNumber=2
{"provingJobId":"1:BASE_PARITY:9e61e5432c1fde3f"}
19:02:04 [19:02:04.994]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1996d3470d4d8525 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1996d3470d4d8525"}
19:02:04 [19:02:04.995]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.996]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:702e99e77edb5602 epochNumber=1
{"provingJobId":"1:BASE_PARITY:702e99e77edb5602"}
19:02:04 [19:02:04.996]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ed6daf69dd8eb8f5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ed6daf69dd8eb8f5"}
19:02:04 [19:02:04.996]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:423cf9da52756d35 epochNumber=2
{"provingJobId":"1:BASE_PARITY:423cf9da52756d35"}
19:02:04 [19:02:04.997]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b88b925ff8953ec4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b88b925ff8953ec4"}
19:02:04 [19:02:04.998]
INFO:
prover-client:proving-broker Proving Broker started
19:02:04 [19:02:04.999]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a0baa0cfed09d108 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a0baa0cfed09d108"}
19:02:04 [19:02:04.999]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b9cb69e60732dd7f epochNumber=1
{"provingJobId":"1:BASE_PARITY:b9cb69e60732dd7f"}
19:02:04 [19:02:04.999]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:85f7eb93fdff63f4 epochNumber=2
{"provingJobId":"1:BASE_PARITY:85f7eb93fdff63f4"}
19:02:04 [19:02:05.000]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8d88c4ccd5757f3f epochNumber=1
{"provingJobId":"1:BASE_PARITY:8d88c4ccd5757f3f"}
19:02:05 [19:02:05.001]
INFO:
prover-client:proving-broker Proving Broker started
19:02:05 [19:02:05.002]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f9ee81b0994b01e epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f9ee81b0994b01e"}
19:02:05 [19:02:05.002]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b271fc74d0aab7bb epochNumber=1
{"provingJobId":"1:BASE_PARITY:b271fc74d0aab7bb"}
19:02:05 [19:02:05.002]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4f9ee81b0994b01e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4f9ee81b0994b01e"}
19:02:05 [19:02:05.004]
INFO:
prover-client:proving-broker Proving Broker started
19:02:05 [19:02:05.004]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c86eee2d4db8a327 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c86eee2d4db8a327"}
19:02:05 [19:02:05.005]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4ef3899a8f0025b epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4ef3899a8f0025b"}
19:02:05 [19:02:05.005]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c86eee2d4db8a327 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:c86eee2d4db8a327"}
19:02:05 [19:02:05.006]
INFO:
prover-client:proving-broker Proving Broker started
19:02:05 [19:02:05.007]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1fed02db548448a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1fed02db548448a5"}
19:02:05 [19:02:05.007]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3c9fb06fe086af7a epochNumber=1
{"provingJobId":"1:BASE_PARITY:3c9fb06fe086af7a"}
19:02:05 [19:02:05.007]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1fed02db548448a5 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:1fed02db548448a5"}
19:02:05 [19:02:05.009]
INFO:
prover-client:proving-broker Proving Broker started
19:02:05 [19:02:05.009]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b56ee626e95a8d55 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b56ee626e95a8d55"}
19:02:05 [19:02:05.010]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b56ee626e95a8d55
{"provingJobId":"1:BASE_PARITY:b56ee626e95a8d55"}
19:02:05 [19:02:05.010]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b56ee626e95a8d55 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:b56ee626e95a8d55"}
19:02:05 [19:02:05.010]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b0558f52e317b8e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b0558f52e317b8e3"}
19:02:05 [19:02:05.011]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b56ee626e95a8d55 has already been completed
{"provingJobId":"1:BASE_PARITY:b56ee626e95a8d55"}
19:02:05 [19:02:05.012]
INFO:
prover-client:proving-broker Proving Broker started
19:02:05 [19:02:05.013]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a042e1dfbe40be0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a042e1dfbe40be0f"}
19:02:05 [19:02:05.013]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:603d8bd56ba9db20 epochNumber=2
{"provingJobId":"1:BASE_PARITY:603d8bd56ba9db20"}
19:02:06 [19:02:06.065]
INFO:
prover-client:proving-broker Proving Broker started
19:02:06 [19:02:06.066]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a042e1dfbe40be0f settled=false
{"provingJobId":"1:BASE_PARITY:a042e1dfbe40be0f","status":"pending"}
19:02:06 [19:02:06.067]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:603d8bd56ba9db20 settled=false
{"provingJobId":"1:BASE_PARITY:603d8bd56ba9db20","status":"pending"}
19:02:06 [19:02:06.068]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a042e1dfbe40be0f type=BASE_PARITY startedAt=1750964525013 older agent has taken job
{"provingJobId":"1:BASE_PARITY:a042e1dfbe40be0f"}
19:02:06 [19:02:06.068]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a042e1dfbe40be0f type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:a042e1dfbe40be0f"}
19:02:06 [19:02:06.069]
INFO:
prover-client:proving-broker Proving Broker started
19:02:06 [19:02:06.070]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be344c9b62a2ad5b epochNumber=1
{"provingJobId":"1:BASE_PARITY:be344c9b62a2ad5b"}
19:02:06 [19:02:06.070]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d46d4d3de5925cf7 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d46d4d3de5925cf7"}
19:02:07 [19:02:07.071]
INFO:
prover-client:proving-broker Proving Broker started
19:02:07 [19:02:07.072]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:be344c9b62a2ad5b settled=false
{"provingJobId":"1:BASE_PARITY:be344c9b62a2ad5b","status":"pending"}
19:02:07 [19:02:07.072]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d46d4d3de5925cf7 settled=false
{"provingJobId":"1:BASE_PARITY:d46d4d3de5925cf7","status":"pending"}
19:02:07 [19:02:07.073]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:be344c9b62a2ad5b type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:be344c9b62a2ad5b"}
19:02:07 [19:02:07.076]
INFO:
prover-client:proving-broker Proving Broker started
19:02:07 [19:02:07.076]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:29102e88ecc106fe epochNumber=1
{"provingJobId":"1:BASE_PARITY:29102e88ecc106fe"}
19:02:07 [19:02:07.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bbe3a298c5159845 epochNumber=2
{"provingJobId":"1:BASE_PARITY:bbe3a298c5159845"}
19:02:08 [19:02:08.078]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.078]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:29102e88ecc106fe settled=false
{"provingJobId":"1:BASE_PARITY:29102e88ecc106fe","status":"pending"}
19:02:08 [19:02:08.078]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bbe3a298c5159845 settled=false
{"provingJobId":"1:BASE_PARITY:bbe3a298c5159845","status":"pending"}
19:02:08 [19:02:08.079]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:29102e88ecc106fe type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:29102e88ecc106fe"}
19:02:08 [19:02:08.079]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:29102e88ecc106fe type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:29102e88ecc106fe"}
19:02:08 [19:02:08.081]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.083]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97b12bbe6db9b96d epochNumber=1
{"provingJobId":"1:BASE_PARITY:97b12bbe6db9b96d"}
19:02:08 [19:02:08.084]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d1cffe4f7b5239d3 epochNumber=2
{"provingJobId":"1:BASE_PARITY:d1cffe4f7b5239d3"}
19:02:08 [19:02:08.084]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:97b12bbe6db9b96d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:97b12bbe6db9b96d"}
19:02:08 [19:02:08.085]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d1cffe4f7b5239d3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:d1cffe4f7b5239d3"}
19:02:08 [19:02:08.086]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba4fa9d836683b3f epochNumber=1
{"provingJobId":"1:BASE_PARITY:ba4fa9d836683b3f"}
19:02:08 [19:02:08.087]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9fad0c4998f504b9 epochNumber=2
{"provingJobId":"1:BASE_PARITY:9fad0c4998f504b9"}
19:02:08 [19:02:08.087]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ba4fa9d836683b3f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ba4fa9d836683b3f"}
19:02:08 [19:02:08.087]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ba4fa9d836683b3f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ba4fa9d836683b3f"}
19:02:08 [19:02:08.088]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9fad0c4998f504b9 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9fad0c4998f504b9"}
19:02:08 [19:02:08.089]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.089]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:02e7b4abf4e7f481 err=test error
{"provingJoId":"1:BASE_PARITY:02e7b4abf4e7f481"}
19:02:08 [19:02:08.090]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.091]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b72d4ecb1997dc4d not found
{"provingJobId":"1:BASE_PARITY:b72d4ecb1997dc4d"}
19:02:08 [19:02:08.092]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.093]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b58bcfa312ad02b epochNumber=1
{"provingJobId":"1:BASE_PARITY:5b58bcfa312ad02b"}
19:02:08 [19:02:08.094]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.094]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1ceba73d2ca16908 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1ceba73d2ca16908"}
19:02:08 [19:02:08.195]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1ceba73d2ca16908 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:1ceba73d2ca16908"}
19:02:08 [19:02:08.198]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.199]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cc5107bc1a79b691 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cc5107bc1a79b691"}
19:02:08 [19:02:08.301]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cc5107bc1a79b691 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:cc5107bc1a79b691"}
19:02:08 [19:02:08.308]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ec074d9efd44ccbb epochNumber=10
{"provingJobId":"1:BASE_PARITY:ec074d9efd44ccbb"}
19:02:08 [19:02:08.328]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:08 [19:02:08.435]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.436]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd4840bc8fcb7034 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd4840bc8fcb7034"}
19:02:08 [19:02:08.591]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd4840bc8fcb7034 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:fd4840bc8fcb7034"}
19:02:08 [19:02:08.639]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.640]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c6bc2ab4f14e12d epochNumber=1
{"provingJobId":"1:BASE_PARITY:0c6bc2ab4f14e12d"}
19:02:08 [19:02:08.641]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0c6bc2ab4f14e12d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0c6bc2ab4f14e12d"}
19:02:08 [19:02:08.642]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.643]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cb123266f8e1a3e6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cb123266f8e1a3e6"}
19:02:08 [19:02:08.647]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cb123266f8e1a3e6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:cb123266f8e1a3e6"}
19:02:08 [19:02:08.648]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cb123266f8e1a3e6 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:cb123266f8e1a3e6"}
19:02:08 [19:02:08.650]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.650]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4cf830790cd87c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e4cf830790cd87c2"}
19:02:08 [19:02:08.651]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e4cf830790cd87c2 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e4cf830790cd87c2"}
19:02:08 [19:02:08.652]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.653]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b82150015816d6ef epochNumber=1
{"provingJobId":"1:BASE_PARITY:b82150015816d6ef"}
19:02:08 [19:02:08.654]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b82150015816d6ef type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:b82150015816d6ef"}
19:02:08 [19:02:08.654]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ba9ad8e7c85016dd epochNumber=3
{"provingJobId":"1:BASE_PARITY:ba9ad8e7c85016dd"}
19:02:08 [19:02:08.678]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:08 [19:02:08.680]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b82150015816d6ef err=test error
{"provingJoId":"1:BASE_PARITY:b82150015816d6ef"}
19:02:08 [19:02:08.683]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.683]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5253c723ed18ff4b settled=false
{"provingJobId":"1:BASE_PARITY:5253c723ed18ff4b","status":"pending"}
19:02:08 [19:02:08.684]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a20c7f6c8427bce4 settled=false
{"provingJobId":"1:BASE_PARITY:a20c7f6c8427bce4","status":"pending"}
19:02:08 [19:02:08.686]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.687]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fbfc3022f79406e3 settled=true
{"provingJobId":"1:BASE_PARITY:fbfc3022f79406e3","status":"fulfilled"}
19:02:08 [19:02:08.687]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:bf42afce9c87f71d settled=true
{"provingJobId":"2:BASE_PARITY:bf42afce9c87f71d","status":"fulfilled"}
19:02:08 [19:02:08.689]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.689]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7017ddb766153aca settled=true
{"provingJobId":"1:BASE_PARITY:7017ddb766153aca","status":"fulfilled"}
19:02:08 [19:02:08.689]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:920dab2ecc67f5c7 settled=false
{"provingJobId":"1:BASE_PARITY:920dab2ecc67f5c7","status":"pending"}
19:02:08 [19:02:08.691]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4c5700f08cd74743 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4c5700f08cd74743"}
19:02:08 [19:02:08.695]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.696]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0544489e95f114a epochNumber=1
{"provingJobId":"1:BASE_PARITY:c0544489e95f114a"}
19:02:08 [19:02:08.698]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:c0544489e95f114a: Error: db error: Error: db error
19:02:08 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:c0544489e95f114a"}
19:02:08 [19:02:08.700]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35ea8ed2b27e24a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:35ea8ed2b27e24a6"}
19:02:08 [19:02:08.701]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:35ea8ed2b27e24a6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:35ea8ed2b27e24a6"}
19:02:08 [19:02:08.701]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:35ea8ed2b27e24a6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:35ea8ed2b27e24a6"}
19:02:08 [19:02:08.703]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.703]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b9c046ad62d161c epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b9c046ad62d161c"}
19:02:08 [19:02:08.703]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3b9c046ad62d161c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3b9c046ad62d161c"}
19:02:08 [19:02:08.704]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3b9c046ad62d161c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3b9c046ad62d161c"}
19:02:08 [19:02:08.704]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:3b9c046ad62d161c: Error: db error
19:02:08 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:3b9c046ad62d161c"}
19:02:08 [19:02:08.706]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.706]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4de3a487cec3659 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f4de3a487cec3659"}
19:02:08 [19:02:08.707]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f4de3a487cec3659 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f4de3a487cec3659"}
19:02:08 [19:02:08.707]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f4de3a487cec3659 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f4de3a487cec3659"}
19:02:08 [19:02:08.709]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.709]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:79a35b2763e335b3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:79a35b2763e335b3"}
19:02:08 [19:02:08.710]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:79a35b2763e335b3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:79a35b2763e335b3"}
19:02:08 [19:02:08.710]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:79a35b2763e335b3 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:79a35b2763e335b3"}
19:02:08 [19:02:08.711]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:79a35b2763e335b3 jobErr=test error: Error: db error
19:02:08 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:79a35b2763e335b3"}
19:02:08 [19:02:08.712]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.713]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:272a7e2b95c124b4 not found
{"provingJobId":"1:BASE_PARITY:272a7e2b95c124b4"}
19:02:08 [19:02:08.714]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.715]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:069da4886dd37f42 err=test error
{"provingJoId":"1:BASE_PARITY:069da4886dd37f42"}
19:02:08 [19:02:08.716]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.726]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d94eef60c3048d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d94eef60c3048d5"}
19:02:08 [19:02:08.727]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3d94eef60c3048d5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:3d94eef60c3048d5"}
19:02:08 [19:02:08.728]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3d94eef60c3048d5 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3d94eef60c3048d5"}
19:02:08 [19:02:08.738]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:2b92bd6de3719de6 epochNumber=2
{"provingJobId":"2:BASE_PARITY:2b92bd6de3719de6"}
19:02:08 [19:02:08.738]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:2b92bd6de3719de6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:2b92bd6de3719de6"}
19:02:08 [19:02:08.738]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2b92bd6de3719de6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:2b92bd6de3719de6"}
19:02:08 [19:02:08.800]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:075f939ccc20cff2 epochNumber=3
{"provingJobId":"3:BASE_PARITY:075f939ccc20cff2"}
19:02:08 [19:02:08.818]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:08 [19:02:08.866]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:dfd97650e8ac9f32 epochNumber=4
{"provingJobId":"4:BASE_PARITY:dfd97650e8ac9f32"}
19:02:08 [19:02:08.872]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:08 [19:02:08.928]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:d7f9923dce55cc1e epochNumber=5
{"provingJobId":"5:BASE_PARITY:d7f9923dce55cc1e"}
19:02:08 [19:02:08.950]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:08 [19:02:08.983]
INFO:
prover-client:proving-broker Proving Broker started
19:02:08 [19:02:08.984]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c0afaf04cde6817c epochNumber=42
{"provingJobId":"1:BASE_PARITY:c0afaf04cde6817c"}
19:02:08 [19:02:08.985]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testprIT3w/42 with map size 1073741824
19:02:08 [19:02:08.990]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.018]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:165e0c480c58e9f1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:165e0c480c58e9f1"}
19:02:09 [19:02:09.025]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.026]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09dd50559dfe6d01 epochNumber=1
{"provingJobId":"1:BASE_PARITY:09dd50559dfe6d01"}
19:02:09 [19:02:09.027]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuOgydD/1 with map size 1073741824
19:02:09 [19:02:09.031]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.056]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d7c388182f7c061 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d7c388182f7c061"}
19:02:09 [19:02:09.072]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.073]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbcc97bb49c9510f epochNumber=1
{"provingJobId":"1:BASE_PARITY:dbcc97bb49c9510f"}
19:02:09 [19:02:09.073]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3RVmIh/1 with map size 1073741824
19:02:09 [19:02:09.078]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.098]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:dbcc97bb49c9510f epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:dbcc97bb49c9510f"}
19:02:09 [19:02:09.102]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.103]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7cbfab878cee46ed epochNumber=1
{"provingJobId":"1:BASE_PARITY:7cbfab878cee46ed"}
19:02:09 [19:02:09.104]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhRwDk3/1 with map size 1073741824
19:02:09 [19:02:09.107]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.123]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7cbfab878cee46ed epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7cbfab878cee46ed"}
19:02:09 [19:02:09.124]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7cbfab878cee46ed type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:7cbfab878cee46ed"}
19:02:09 [19:02:09.130]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7cbfab878cee46ed epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:7cbfab878cee46ed"}
19:02:09 [19:02:09.138]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.138]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:970e2513d523ed40 epochNumber=1
{"provingJobId":"1:BASE_PARITY:970e2513d523ed40"}
19:02:09 [19:02:09.139]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmZ1m17/1 with map size 1073741824
19:02:09 [19:02:09.142]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.174]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.181]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.183]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24798b332f5440c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:24798b332f5440c6"}
19:02:09 [19:02:09.184]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzY71bo/1 with map size 1073741824
19:02:09 [19:02:09.189]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.211]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:24798b332f5440c6
{"provingJobId":"1:BASE_PARITY:24798b332f5440c6"}
19:02:09 [19:02:09.212]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:24798b332f5440c6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:24798b332f5440c6"}
19:02:09 [19:02:09.212]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:24798b332f5440c6 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:24798b332f5440c6"}
19:02:09 [19:02:09.225]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.226]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f9f82a2e3194a43d epochNumber=1
{"provingJobId":"1:BASE_PARITY:f9f82a2e3194a43d"}
19:02:09 [19:02:09.227]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNunENA/1 with map size 1073741824
19:02:09 [19:02:09.231]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.256]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f9f82a2e3194a43d
{"provingJobId":"1:BASE_PARITY:f9f82a2e3194a43d"}
19:02:09 [19:02:09.257]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f9f82a2e3194a43d type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f9f82a2e3194a43d"}
19:02:09 [19:02:09.266]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.266]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:46c48461066ee58e epochNumber=1
{"provingJobId":"1:BASE_PARITY:46c48461066ee58e"}
19:02:09 [19:02:09.267]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcNACcE/1 with map size 1073741824
19:02:09 [19:02:09.273]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.294]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:46c48461066ee58e type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:46c48461066ee58e"}
19:02:09 [19:02:09.294]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:46c48461066ee58e type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:46c48461066ee58e"}
19:02:09 [19:02:09.307]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.308]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9dac0baf6678f1ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:9dac0baf6678f1ea"}
19:02:09 [19:02:09.311]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6bLlG6/1 with map size 1073741824
19:02:09 [19:02:09.318]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.341]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9dac0baf6678f1ea type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9dac0baf6678f1ea"}
19:02:09 [19:02:09.341]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9dac0baf6678f1ea type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9dac0baf6678f1ea"}
19:02:09 [19:02:09.359]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.360]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0ae80db04248f138 epochNumber=0
{"provingJobId":"1:BASE_PARITY:0ae80db04248f138"}
19:02:09 [19:02:09.361]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testA0oP9s/0 with map size 1073741824
19:02:09 [19:02:09.372]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.402]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0ae80db04248f138 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0ae80db04248f138"}
19:02:09 [19:02:09.403]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0e41e503e88d89df epochNumber=0
{"provingJobId":"1:BASE_PARITY:0e41e503e88d89df"}
19:02:09 [19:02:09.404]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0ae80db04248f138 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:0ae80db04248f138"}
19:02:09 [19:02:09.404]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e061076b8610544 epochNumber=0
{"provingJobId":"1:BASE_PARITY:6e061076b8610544"}
19:02:09 [19:02:09.430]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.433]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.434]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa1a6925779c01a1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:fa1a6925779c01a1"}
19:02:09 [19:02:09.435]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFNQ75z/2 with map size 1073741824
19:02:09 [19:02:09.439]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.464]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ebf89b83b463d5c2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ebf89b83b463d5c2"}
19:02:09 [19:02:09.466]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFNQ75z/1 with map size 1073741824
19:02:09 [19:02:09.470]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.494]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.494]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa76c398aa3b8d2f epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa76c398aa3b8d2f"}
19:02:09 [19:02:09.495]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1S756A/1 with map size 1073741824
19:02:09 [19:02:09.498]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.520]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.521]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f9a12c048abd08b epochNumber=1
{"provingJobId":"1:BASE_PARITY:2f9a12c048abd08b"}
19:02:09 [19:02:09.522]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwVPIba/1 with map size 1073741824
19:02:09 [19:02:09.524]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ef17685e47a45e37 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ef17685e47a45e37"}
19:02:09 [19:02:09.555]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f1b0eacabfcd8628 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f1b0eacabfcd8628"}
19:02:09 [19:02:09.556]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwVPIba/2 with map size 1073741824
19:02:09 [19:02:09.558]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.581]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:454ba871ed9387fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:454ba871ed9387fb"}
19:02:09 [19:02:09.593]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.594]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:df59923ea031a75e epochNumber=1
{"provingJobId":"1:BASE_PARITY:df59923ea031a75e"}
19:02:09 [19:02:09.595]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXscmgj/1 with map size 1073741824
19:02:09 [19:02:09.597]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3a7d5ee902718e3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3a7d5ee902718e3"}
19:02:09 [19:02:09.625]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:970f6d84f34006ca epochNumber=2
{"provingJobId":"1:BASE_PARITY:970f6d84f34006ca"}
19:02:09 [19:02:09.626]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testXscmgj/2 with map size 1073741824
19:02:09 [19:02:09.628]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.647]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16f1d4035f0bde66 epochNumber=1
{"provingJobId":"1:BASE_PARITY:16f1d4035f0bde66"}
19:02:09 [19:02:09.658]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.659]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dec40af3c276b624 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dec40af3c276b624"}
19:02:09 [19:02:09.660]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5ZV2Q5/1 with map size 1073741824
19:02:09 [19:02:09.673]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f4a247af2f8c07d epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f4a247af2f8c07d"}
19:02:09 [19:02:09.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:84c7ec857d44e4af epochNumber=2
{"provingJobId":"1:BASE_PARITY:84c7ec857d44e4af"}
19:02:09 [19:02:09.701]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test5ZV2Q5/2 with map size 1073741824
19:02:09 [19:02:09.705]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.721]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:31726df5d7980810 epochNumber=1
{"provingJobId":"1:BASE_PARITY:31726df5d7980810"}
19:02:09 [19:02:09.733]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.734]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:329c0894e0a1184f epochNumber=1
{"provingJobId":"1:BASE_PARITY:329c0894e0a1184f"}
19:02:09 [19:02:09.736]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjdPgI1/1 with map size 1073741824
19:02:09 [19:02:09.741]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.766]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:38fbaeadb42a0051 epochNumber=1
{"provingJobId":"1:BASE_PARITY:38fbaeadb42a0051"}
19:02:09 [19:02:09.772]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:329c0894e0a1184f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:329c0894e0a1184f"}
19:02:09 [19:02:09.780]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.781]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ca37e35630cc8e34 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ca37e35630cc8e34"}
19:02:09 [19:02:09.782]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFTA1ub/1 with map size 1073741824
19:02:09 [19:02:09.783]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.801]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a5547d40696145f epochNumber=1
{"provingJobId":"1:BASE_PARITY:0a5547d40696145f"}
19:02:09 [19:02:09.807]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ca37e35630cc8e34 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:ca37e35630cc8e34"}
19:02:09 [19:02:09.819]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.820]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f7431f848c1817f epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f7431f848c1817f"}
19:02:09 [19:02:09.821]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrG4XF1/1 with map size 1073741824
19:02:09 [19:02:09.824]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.844]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b2d18b1f3b3b24e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b2d18b1f3b3b24e1"}
19:02:09 [19:02:09.858]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9f7431f848c1817f type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:9f7431f848c1817f"}
19:02:09 [19:02:09.863]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.864]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aa1ba9226ba486d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:aa1ba9226ba486d1"}
19:02:09 [19:02:09.865]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9L9upH/1 with map size 1073741824
19:02:09 [19:02:09.867]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.887]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:aa1ba9226ba486d1
{"provingJobId":"1:BASE_PARITY:aa1ba9226ba486d1"}
19:02:09 [19:02:09.888]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:aa1ba9226ba486d1 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:aa1ba9226ba486d1"}
19:02:09 [19:02:09.894]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:626da5ed0420d875 epochNumber=1
{"provingJobId":"1:BASE_PARITY:626da5ed0420d875"}
19:02:09 [19:02:09.900]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aa1ba9226ba486d1 has already been completed
{"provingJobId":"1:BASE_PARITY:aa1ba9226ba486d1"}
19:02:09 [19:02:09.907]
INFO:
prover-client:proving-broker Proving Broker started
19:02:09 [19:02:09.909]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c81489a03746f661 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c81489a03746f661"}
19:02:09 [19:02:09.910]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpQggfg/1 with map size 1073741824
19:02:09 [19:02:09.913]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:09 [19:02:09.933]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:24ee2ae98fc13874 epochNumber=2
{"provingJobId":"1:BASE_PARITY:24ee2ae98fc13874"}
19:02:09 [19:02:09.934]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testpQggfg/2 with map size 1073741824
19:02:09 [19:02:09.935]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:11 [19:02:11.024]
INFO:
prover-client:proving-broker Proving Broker started
19:02:11 [19:02:11.031]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c81489a03746f661 settled=false
{"provingJobId":"1:BASE_PARITY:c81489a03746f661","status":"pending"}
19:02:11 [19:02:11.033]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:24ee2ae98fc13874 settled=false
{"provingJobId":"1:BASE_PARITY:24ee2ae98fc13874","status":"pending"}
19:02:11 [19:02:11.035]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c81489a03746f661 type=BASE_PARITY startedAt=1750964529973 older agent has taken job
{"provingJobId":"1:BASE_PARITY:c81489a03746f661"}
19:02:11 [19:02:11.035]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c81489a03746f661 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:c81489a03746f661"}
19:02:11 [19:02:11.040]
INFO:
prover-client:proving-broker Proving Broker started
19:02:11 [19:02:11.041]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b83e46e61503ff9e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b83e46e61503ff9e"}
19:02:11 [19:02:11.042]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8OiAIl/1 with map size 1073741824
19:02:11 [19:02:11.047]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:11 [19:02:11.066]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a361b9f2daead8c2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a361b9f2daead8c2"}
19:02:11 [19:02:11.067]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8OiAIl/2 with map size 1073741824
19:02:11 [19:02:11.070]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:12 [19:02:12.092]
INFO:
prover-client:proving-broker Proving Broker started
19:02:12 [19:02:12.095]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b83e46e61503ff9e settled=false
{"provingJobId":"1:BASE_PARITY:b83e46e61503ff9e","status":"pending"}
19:02:12 [19:02:12.098]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a361b9f2daead8c2 settled=false
{"provingJobId":"1:BASE_PARITY:a361b9f2daead8c2","status":"pending"}
19:02:12 [19:02:12.100]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b83e46e61503ff9e type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:b83e46e61503ff9e"}
19:02:12 [19:02:12.106]
INFO:
prover-client:proving-broker Proving Broker started
19:02:12 [19:02:12.107]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21029c4421210fde epochNumber=1
{"provingJobId":"1:BASE_PARITY:21029c4421210fde"}
19:02:12 [19:02:12.108]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTUOgzL/1 with map size 1073741824
19:02:12 [19:02:12.114]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:12 [19:02:12.130]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ba70e5bb608ae25 epochNumber=2
{"provingJobId":"1:BASE_PARITY:5ba70e5bb608ae25"}
19:02:12 [19:02:12.132]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTUOgzL/2 with map size 1073741824
19:02:12 [19:02:12.134]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.158]
INFO:
prover-client:proving-broker Proving Broker started
19:02:13 [19:02:13.162]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:21029c4421210fde settled=false
{"provingJobId":"1:BASE_PARITY:21029c4421210fde","status":"pending"}
19:02:13 [19:02:13.164]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5ba70e5bb608ae25 settled=false
{"provingJobId":"1:BASE_PARITY:5ba70e5bb608ae25","status":"pending"}
19:02:13 [19:02:13.165]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:21029c4421210fde type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:21029c4421210fde"}
19:02:13 [19:02:13.166]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:21029c4421210fde type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:21029c4421210fde"}
19:02:13 [19:02:13.178]
INFO:
prover-client:proving-broker Proving Broker started
19:02:13 [19:02:13.179]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:63251263b65b2a9b epochNumber=1
{"provingJobId":"1:BASE_PARITY:63251263b65b2a9b"}
19:02:13 [19:02:13.180]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8YsZ7D/1 with map size 1073741824
19:02:13 [19:02:13.184]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:22cef93dd8966039 epochNumber=2
{"provingJobId":"1:BASE_PARITY:22cef93dd8966039"}
19:02:13 [19:02:13.201]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8YsZ7D/2 with map size 1073741824
19:02:13 [19:02:13.208]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.227]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:63251263b65b2a9b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:63251263b65b2a9b"}
19:02:13 [19:02:13.234]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:22cef93dd8966039 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:22cef93dd8966039"}
19:02:13 [19:02:13.250]
INFO:
prover-client:proving-broker Proving Broker started
19:02:13 [19:02:13.252]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f66dfec47f322317 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f66dfec47f322317"}
19:02:13 [19:02:13.257]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testo4spgh/1 with map size 1073741824
19:02:13 [19:02:13.267]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.287]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6407eeaa88047f7a epochNumber=2
{"provingJobId":"1:BASE_PARITY:6407eeaa88047f7a"}
19:02:13 [19:02:13.288]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testo4spgh/2 with map size 1073741824
19:02:13 [19:02:13.290]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.313]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f66dfec47f322317 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f66dfec47f322317"}
19:02:13 [19:02:13.313]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f66dfec47f322317 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:f66dfec47f322317"}
19:02:13 [19:02:13.319]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6407eeaa88047f7a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6407eeaa88047f7a"}
19:02:13 [19:02:13.329]
INFO:
prover-client:proving-broker Proving Broker started
19:02:13 [19:02:13.329]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ea4dfc5e07c8e955 err=test error
{"provingJoId":"1:BASE_PARITY:ea4dfc5e07c8e955"}
19:02:13 [19:02:13.332]
INFO:
prover-client:proving-broker Proving Broker started
19:02:13 [19:02:13.333]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3d2bba2a13f45729 not found
{"provingJobId":"1:BASE_PARITY:3d2bba2a13f45729"}
19:02:13 [19:02:13.336]
INFO:
prover-client:proving-broker Proving Broker started
19:02:13 [19:02:13.336]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:665ecfea47922a1c epochNumber=1
{"provingJobId":"1:BASE_PARITY:665ecfea47922a1c"}
19:02:13 [19:02:13.337]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxbdUv2/1 with map size 1073741824
19:02:13 [19:02:13.339]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.364]
INFO:
prover-client:proving-broker Proving Broker started
19:02:13 [19:02:13.366]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52942273bcea2651 epochNumber=1
{"provingJobId":"1:BASE_PARITY:52942273bcea2651"}
19:02:13 [19:02:13.370]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSrC8AB/1 with map size 1073741824
19:02:13 [19:02:13.373]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.493]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:52942273bcea2651 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:52942273bcea2651"}
19:02:13 [19:02:13.501]
INFO:
prover-client:proving-broker Proving Broker started
19:02:13 [19:02:13.502]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9dc103dd22a3c919 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9dc103dd22a3c919"}
19:02:13 [19:02:13.504]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testL1I4pK/1 with map size 1073741824
19:02:13 [19:02:13.508]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.630]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9dc103dd22a3c919 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:9dc103dd22a3c919"}
19:02:13 [19:02:13.636]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e29a656c87708698 epochNumber=10
{"provingJobId":"1:BASE_PARITY:e29a656c87708698"}
19:02:13 [19:02:13.636]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testL1I4pK/10 with map size 1073741824
19:02:13 [19:02:13.639]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.681]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:13 [19:02:13.682]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:02:13 [19:02:13.796]
INFO:
prover-client:proving-broker Proving Broker started
19:02:13 [19:02:13.798]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d419b1bb30ca617b epochNumber=1
{"provingJobId":"1:BASE_PARITY:d419b1bb30ca617b"}
19:02:13 [19:02:13.798]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8qBe8H/1 with map size 1073741824
19:02:13 [19:02:13.801]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:13 [19:02:13.976]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:d419b1bb30ca617b timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:d419b1bb30ca617b"}
19:02:14 [19:02:14.028]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:60f91ad1bb44fdcd epochNumber=1
{"provingJobId":"1:BASE_PARITY:60f91ad1bb44fdcd"}
19:02:14 [19:02:14.030]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGgKxLp/1 with map size 1073741824
19:02:14 [19:02:14.033]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.054]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:60f91ad1bb44fdcd type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:60f91ad1bb44fdcd"}
19:02:14 [19:02:14.068]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.069]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a52354662efdec52 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a52354662efdec52"}
19:02:14 [19:02:14.070]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlMANZO/1 with map size 1073741824
19:02:14 [19:02:14.073]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.093]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a52354662efdec52 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:a52354662efdec52"}
19:02:14 [19:02:14.094]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a52354662efdec52 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:a52354662efdec52"}
19:02:14 [19:02:14.105]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.107]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eed72f3f8c0b1a36 epochNumber=1
{"provingJobId":"1:BASE_PARITY:eed72f3f8c0b1a36"}
19:02:14 [19:02:14.108]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSX3W4k/1 with map size 1073741824
19:02:14 [19:02:14.110]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.133]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:eed72f3f8c0b1a36 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:eed72f3f8c0b1a36"}
19:02:14 [19:02:14.151]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.153]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0475df981905a7f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0475df981905a7f2"}
19:02:14 [19:02:14.154]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGBLAGe/1 with map size 1073741824
19:02:14 [19:02:14.158]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.178]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0475df981905a7f2 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0475df981905a7f2"}
19:02:14 [19:02:14.179]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:41b3c5b46ac6dc62 epochNumber=3
{"provingJobId":"1:BASE_PARITY:41b3c5b46ac6dc62"}
19:02:14 [19:02:14.182]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testGBLAGe/3 with map size 1073741824
19:02:14 [19:02:14.185]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.228]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:14 [19:02:14.228]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:02:14 [19:02:14.229]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0475df981905a7f2 err=test error
{"provingJoId":"1:BASE_PARITY:0475df981905a7f2"}
19:02:14 [19:02:14.234]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test10VSLH/1 with map size 1073741824
19:02:14 [19:02:14.236]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.257]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test10VSLH/2 with map size 1073741824
19:02:14 [19:02:14.265]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.281]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.284]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5d3120abe8e78d45 settled=false
{"provingJobId":"1:BASE_PARITY:5d3120abe8e78d45","status":"pending"}
19:02:14 [19:02:14.285]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ffbad2c66f606589 settled=false
{"provingJobId":"1:BASE_PARITY:ffbad2c66f606589","status":"pending"}
19:02:14 [19:02:14.293]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIQ4f2z/1 with map size 1073741824
19:02:14 [19:02:14.295]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.312]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIQ4f2z/2 with map size 1073741824
19:02:14 [19:02:14.315]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.343]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.350]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:96e0dc1518c0372e settled=true
{"provingJobId":"1:BASE_PARITY:96e0dc1518c0372e","status":"fulfilled"}
19:02:14 [19:02:14.367]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b6b30bf4e15ba8af settled=true
{"provingJobId":"2:BASE_PARITY:b6b30bf4e15ba8af","status":"fulfilled"}
19:02:14 [19:02:14.374]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqZBDaw/1 with map size 1073741824
19:02:14 [19:02:14.379]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.402]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqZBDaw/2 with map size 1073741824
19:02:14 [19:02:14.406]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.423]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.426]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5d1d0e3c8bf267d5 settled=true
{"provingJobId":"1:BASE_PARITY:5d1d0e3c8bf267d5","status":"fulfilled"}
19:02:14 [19:02:14.428]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:528c6866e67997c9 settled=false
{"provingJobId":"1:BASE_PARITY:528c6866e67997c9","status":"pending"}
19:02:14 [19:02:14.433]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.433]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a1c0064aad8be362 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a1c0064aad8be362"}
19:02:14 [19:02:14.434]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnayRLk/1 with map size 1073741824
19:02:14 [19:02:14.436]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.465]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.466]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2cc42208d4cc4207 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2cc42208d4cc4207"}
19:02:14 [19:02:14.466]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:2cc42208d4cc4207: Error: db error: Error: db error
19:02:14 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:2cc42208d4cc4207"}
19:02:14 [19:02:14.470]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.471]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58ae12f0afd8df2d epochNumber=1
{"provingJobId":"1:BASE_PARITY:58ae12f0afd8df2d"}
19:02:14 [19:02:14.472]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test34exz6/1 with map size 1073741824
19:02:14 [19:02:14.480]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.507]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:58ae12f0afd8df2d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:58ae12f0afd8df2d"}
19:02:14 [19:02:14.508]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:58ae12f0afd8df2d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:58ae12f0afd8df2d"}
19:02:14 [19:02:14.522]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.523]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:50471c254020d192 epochNumber=1
{"provingJobId":"1:BASE_PARITY:50471c254020d192"}
19:02:14 [19:02:14.523]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIXccmC/1 with map size 1073741824
19:02:14 [19:02:14.527]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.546]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:50471c254020d192 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:50471c254020d192"}
19:02:14 [19:02:14.546]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:50471c254020d192 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:50471c254020d192"}
19:02:14 [19:02:14.547]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:50471c254020d192: Error: db error
19:02:14 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:50471c254020d192"}
19:02:14 [19:02:14.558]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.559]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0bbb2daae6c1cf1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0bbb2daae6c1cf1"}
19:02:14 [19:02:14.562]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwjZtwn/1 with map size 1073741824
19:02:14 [19:02:14.565]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.592]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e0bbb2daae6c1cf1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e0bbb2daae6c1cf1"}
19:02:14 [19:02:14.593]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e0bbb2daae6c1cf1 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:e0bbb2daae6c1cf1"}
19:02:14 [19:02:14.605]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.606]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6f5b3a376c7cbbe8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6f5b3a376c7cbbe8"}
19:02:14 [19:02:14.608]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIVlkpK/1 with map size 1073741824
19:02:14 [19:02:14.612]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.634]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6f5b3a376c7cbbe8 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6f5b3a376c7cbbe8"}
19:02:14 [19:02:14.635]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6f5b3a376c7cbbe8 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6f5b3a376c7cbbe8"}
19:02:14 [19:02:14.636]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:6f5b3a376c7cbbe8 jobErr=test error: Error: db error
19:02:14 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:6f5b3a376c7cbbe8"}
19:02:14 [19:02:14.643]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.643]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:feae204f62b71c77 not found
{"provingJobId":"1:BASE_PARITY:feae204f62b71c77"}
19:02:14 [19:02:14.652]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.653]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:fe0740fe323019aa err=test error
{"provingJoId":"1:BASE_PARITY:fe0740fe323019aa"}
19:02:14 [19:02:14.668]
INFO:
prover-client:proving-broker Proving Broker started
19:02:14 [19:02:14.679]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:704ab7acda52b95c epochNumber=1
{"provingJobId":"1:BASE_PARITY:704ab7acda52b95c"}
19:02:14 [19:02:14.680]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf6Ngrb/1 with map size 1073741824
19:02:14 [19:02:14.684]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.708]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:704ab7acda52b95c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:704ab7acda52b95c"}
19:02:14 [19:02:14.709]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:704ab7acda52b95c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:704ab7acda52b95c"}
19:02:14 [19:02:14.727]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:404f914515761a98 epochNumber=2
{"provingJobId":"2:BASE_PARITY:404f914515761a98"}
19:02:14 [19:02:14.729]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testf6Ngrb/2 with map size 1073741824
19:02:14 [19:02:14.732]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.756]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:404f914515761a98 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:404f914515761a98"}
19:02:14 [19:02:14.757]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:404f914515761a98 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:404f914515761a98"}
19:02:14 [19:02:14.826]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:4a10dd2bba1e3172 epochNumber=3
{"provingJobId":"3:BASE_PARITY:4a10dd2bba1e3172"}
19:02:14 [19:02:14.827]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testf6Ngrb/3 with map size 1073741824
19:02:14 [19:02:14.830]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.853]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:14 [19:02:14.854]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
19:02:14 [19:02:14.915]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:fbcab63e2178de19 epochNumber=4
{"provingJobId":"4:BASE_PARITY:fbcab63e2178de19"}
19:02:14 [19:02:14.917]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testf6Ngrb/4 with map size 1073741824
19:02:14 [19:02:14.921]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:14 [19:02:14.958]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:14 [19:02:14.959]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
19:02:15 [19:02:15.004]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:05a0bd9371f9ef68 epochNumber=5
{"provingJobId":"5:BASE_PARITY:05a0bd9371f9ef68"}
19:02:15 [19:02:15.005]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testf6Ngrb/5 with map size 1073741824
19:02:15 [19:02:15.009]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
19:02:15 [19:02:15.039]
INFO:
prover-client:proving-broker Cleaned up jobs=1
19:02:15 [19:02:15.040]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
19:02:15
PASS src/proving_broker/proving_broker.test.ts (
13.725 s)
19:02:15 ProvingBroker
19:02:15 Producer API
19:02:15
✓ refuses stale jobs (51 ms)
19:02:15
✓ enqueues jobs (4 ms)
19:02:15
✓ ignores duplicate jobs (4 ms)
19:02:15
✓ reports correct status when enqueuing repeat jobs (4 ms)
19:02:15
✓ throws an error in case of duplicate job IDs (29 ms)
19:02:15
✓ returns not-found status for non-existing jobs (3 ms)
19:02:15
✓ cancels jobs in queue (5 ms)
19:02:15
✓ cancels jobs in-progress (12 ms)
19:02:15
✓ returns job result if successful (5 ms)
19:02:15
✓ returns job error if failed (3 ms)
19:02:15
✓ correctly returns job status for concurrent writes (3 ms)
19:02:15
✓ refuses stale jobs (42 ms)
19:02:15
✓ enqueues jobs (46 ms)
19:02:15
✓ ignores duplicate jobs (30 ms)
19:02:15
✓ reports correct status when enqueuing repeat jobs (34 ms)
19:02:15
✓ throws an error in case of duplicate job IDs (32 ms)
19:02:15
✓ returns not-found status for non-existing jobs (7 ms)
19:02:15
✓ cancels jobs in queue (46 ms)
19:02:15
✓ cancels jobs in-progress (40 ms)
19:02:15
✓ returns job result if successful (41 ms)
19:02:15
✓ returns job error if failed (48 ms)
19:02:15
✓ correctly returns job status for concurrent writes (72 ms)
19:02:15 Consumer API
19:02:15
✓ returns undefined if no jobs are available (2 ms)
19:02:15
✓ returns jobs in priority order (2 ms)
19:02:15
✓ returns undefined if no jobs are available for the given allowList (2 ms)
19:02:15
✓ returns a job if it is in the allowList (3 ms)
19:02:15
✓ returns the most important job if it is in the allowList (2 ms)
19:02:15
✓ returns any job if filter is empty (2 ms)
19:02:15
✓ returns a new job when reporting job success (3 ms)
19:02:15
✓ returns a new job when reporting permanent error (2 ms)
19:02:15
✓ returns a new job when reporting retry-able error (2 ms)
19:02:15
✓ returns a new job when reporting progress if current one is cancelled (4 ms)
19:02:15
✓ returns a new job if job is already in progress elsewhere (1057 ms)
19:02:15
✓ avoids sending the same job to a new agent after a restart (1006 ms)
19:02:15
✓ avoids sending a completed job to a new agent after a restart (1005 ms)
19:02:15
✓ tracks job result if in progress (5 ms)
19:02:15
✓ tracks job result even if job is in queue (2 ms)
19:02:15
✓ ignores reported job error if unknown job (2 ms)
19:02:15
✓ ignores job result if unknown job (1 ms)
19:02:15
✓ returns undefined if no jobs are available (5 ms)
19:02:15
✓ returns jobs in priority order (60 ms)
19:02:15
✓ returns undefined if no jobs are available for the given allowList (26 ms)
19:02:15
✓ returns a job if it is in the allowList (72 ms)
19:02:15
✓ returns the most important job if it is in the allowList (65 ms)
19:02:15
✓ returns any job if filter is empty (74 ms)
19:02:15
✓ returns a new job when reporting job success (49 ms)
19:02:15
✓ returns a new job when reporting permanent error (36 ms)
19:02:15
✓ returns a new job when reporting retry-able error (45 ms)
19:02:15
✓ returns a new job when reporting progress if current one is cancelled (41 ms)
19:02:15
✓ returns a new job if job is already in progress elsewhere (1134 ms)
19:02:15
✓ avoids sending the same job to a new agent after a restart (1066 ms)
19:02:15
✓ avoids sending a completed job to a new agent after a restart (1071 ms)
19:02:15
✓ tracks job result if in progress (66 ms)
19:02:15
✓ tracks job result even if job is in queue (83 ms)
19:02:15
✓ ignores reported job error if unknown job (3 ms)
19:02:15
✓ ignores job result if unknown job (3 ms)
19:02:15 Timeouts
19:02:15
✓ tracks in progress jobs (1 ms)
19:02:15
✓ re-enqueues jobs that time out (103 ms)
19:02:15
✓ cancel stale jobs that time out (237 ms)
19:02:15
✓ keeps the jobs in progress while it is alive (203 ms)
19:02:15
✓ tracks in progress jobs (27 ms)
19:02:15
✓ re-enqueues jobs that time out (136 ms)
19:02:15
✓ cancel stale jobs that time out (294 ms)
19:02:15
✓ keeps the jobs in progress while it is alive (233 ms)
19:02:15 Retries
19:02:15
✓ retries jobs (4 ms)
19:02:15
✓ retries up to a maximum number of times (7 ms)
19:02:15
✓ passing retry=false does not retry (3 ms)
19:02:15
✓ does not retry if job is stale (29 ms)
19:02:15
✓ retries jobs (38 ms)
19:02:15
✓ retries up to a maximum number of times (36 ms)
19:02:15
✓ passing retry=false does not retry (45 ms)
19:02:15
✓ does not retry if job is stale (84 ms)
19:02:15 Database management
19:02:15
✓ re-enqueues proof requests on start (3 ms)
19:02:15
✓ restores proof results on start (2 ms)
19:02:15
✓ only re-enqueues unfinished jobs (2 ms)
19:02:15
✓ saves job when enqueued (4 ms)
19:02:15
✓ does not retain job if database fails to save (4 ms)
19:02:15
✓ saves job result (3 ms)
19:02:15
✓ saves result even if database fails to save (3 ms)
19:02:15
✓ saves job error (3 ms)
19:02:15
✓ saves job error even if database fails to save (3 ms)
19:02:15
✓ does not save job result if job is unknown (2 ms)
19:02:15
✓ does not save job error if job is unknown (1 ms)
19:02:15
✓ cleans up old jobs periodically (264 ms)
19:02:15
✓ re-enqueues proof requests on start (58 ms)
19:02:15
✓ restores proof results on start (82 ms)
19:02:15
✓ only re-enqueues unfinished jobs (60 ms)
19:02:15
✓ saves job when enqueued (30 ms)
19:02:15
✓ does not retain job if database fails to save (5 ms)
19:02:15
✓ saves job result (50 ms)
19:02:15
✓ saves result even if database fails to save (32 ms)
19:02:15
✓ saves job error (53 ms)
19:02:15
✓ saves job error even if database fails to save (35 ms)
19:02:15
✓ does not save job result if job is unknown (6 ms)
19:02:15
✓ does not save job error if job is unknown (10 ms)
19:02:15
✓ cleans up old jobs periodically (427 ms)
19:02:15
19:02:15
Test Suites: 1 passed, 1 total
19:02:15
Tests: 96 passed, 96 total
19:02:15
Snapshots: 0 total
19:02:15
Time: 13.832 s
19:02:15
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
19:02:15
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?