Parent Log: http://ci.aztec-labs.com/0d8f7888711d05bb Command: 6e611b2acba6dc44 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/9a37be7dc4208f0eeb44ec50194c00ca3f5c4cc6 Env: REF_NAME=gh-readonly-queue/next/pr-15072-1e338a3fb2e2077f1feaee8b86c42644ff8a5352 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Mon Jun 16 16:41:49 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15072_amd64_x2-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 16:41:53 [16:41:53.493] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.500] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ddd0b003d17603d7 epochNumber=42 {"provingJobId":"1:BASE_PARITY:ddd0b003d17603d7"} 16:41:53 [16:41:53.504] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:4ebf0f674034bd88 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4ebf0f674034bd88"} 16:41:53 [16:41:53.543] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.544] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6cd58ebb03f78e13 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6cd58ebb03f78e13"} 16:41:53 [16:41:53.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:af639f4efc15ad15 epochNumber=1 {"provingJobId":"1:BASE_PARITY:af639f4efc15ad15"} 16:41:53 [16:41:53.548] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.549] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c92bf95a7ffd58b3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c92bf95a7ffd58b3"} 16:41:53 [16:41:53.550] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c92bf95a7ffd58b3 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:c92bf95a7ffd58b3"} 16:41:53 [16:41:53.552] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.553] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7310340433575aeb epochNumber=1 {"provingJobId":"1:BASE_PARITY:7310340433575aeb"} 16:41:53 [16:41:53.554] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7310340433575aeb epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:7310340433575aeb"} 16:41:53 [16:41:53.555] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7310340433575aeb type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:7310340433575aeb"} 16:41:53 [16:41:53.555] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7310340433575aeb epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:7310340433575aeb"} 16:41:53 [16:41:53.557] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.557] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:db02872c0acac96a epochNumber=1 {"provingJobId":"1:BASE_PARITY:db02872c0acac96a"} 16:41:53 [16:41:53.595] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.598] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.599] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0537a3e7d0509748 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0537a3e7d0509748"} 16:41:53 [16:41:53.600] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0537a3e7d0509748 {"provingJobId":"1:BASE_PARITY:0537a3e7d0509748"} 16:41:53 [16:41:53.601] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0537a3e7d0509748 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0537a3e7d0509748"} 16:41:53 [16:41:53.601] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0537a3e7d0509748 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:0537a3e7d0509748"} 16:41:53 [16:41:53.603] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.604] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:11b4edd7b9f3d1b8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:11b4edd7b9f3d1b8"} 16:41:53 [16:41:53.605] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:11b4edd7b9f3d1b8 {"provingJobId":"1:BASE_PARITY:11b4edd7b9f3d1b8"} 16:41:53 [16:41:53.606] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:11b4edd7b9f3d1b8 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:11b4edd7b9f3d1b8"} 16:41:53 [16:41:53.608] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.610] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8c1ea58ba7128a75 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8c1ea58ba7128a75"} 16:41:53 [16:41:53.611] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8c1ea58ba7128a75 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8c1ea58ba7128a75"} 16:41:53 [16:41:53.611] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8c1ea58ba7128a75 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8c1ea58ba7128a75"} 16:41:53 [16:41:53.614] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.615] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:985c3478e913e37e epochNumber=1 {"provingJobId":"1:BASE_PARITY:985c3478e913e37e"} 16:41:53 [16:41:53.616] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:985c3478e913e37e type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:985c3478e913e37e"} 16:41:53 [16:41:53.616] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:985c3478e913e37e type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:985c3478e913e37e"} 16:41:53 [16:41:53.618] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.619] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:398f1ff8b9aaea78 epochNumber=0 {"provingJobId":"1:BASE_PARITY:398f1ff8b9aaea78"} 16:41:53 [16:41:53.619] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:398f1ff8b9aaea78 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:398f1ff8b9aaea78"} 16:41:53 [16:41:53.619] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:128c3b9e73eed860 epochNumber=0 {"provingJobId":"1:BASE_PARITY:128c3b9e73eed860"} 16:41:53 [16:41:53.620] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:398f1ff8b9aaea78 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:398f1ff8b9aaea78"} 16:41:53 [16:41:53.620] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:36c79d1cf868c848 epochNumber=0 {"provingJobId":"1:BASE_PARITY:36c79d1cf868c848"} 16:41:53 [16:41:53.621] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.623] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.623] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4a1da9b7f295400e epochNumber=2 {"provingJobId":"1:BASE_PARITY:4a1da9b7f295400e"} 16:41:53 [16:41:53.624] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7c5d01f737cd0694 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7c5d01f737cd0694"} 16:41:53 [16:41:53.625] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.626] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:018704d5975d9398 epochNumber=1 {"provingJobId":"1:BASE_PARITY:018704d5975d9398"} 16:41:53 [16:41:53.627] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.627] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c583d153c07d8fd6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c583d153c07d8fd6"} 16:41:53 [16:41:53.627] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a3160eb1600e5304 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a3160eb1600e5304"} 16:41:53 [16:41:53.628] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:13f76b909614e795 epochNumber=2 {"provingJobId":"1:BASE_PARITY:13f76b909614e795"} 16:41:53 [16:41:53.628] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7f551dc021c2ba45 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7f551dc021c2ba45"} 16:41:53 [16:41:53.629] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.630] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:19437b4c04388c49 epochNumber=1 {"provingJobId":"1:BASE_PARITY:19437b4c04388c49"} 16:41:53 [16:41:53.630] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:38e79b691a40e7c5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:38e79b691a40e7c5"} 16:41:53 [16:41:53.630] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9857d1b8568dcee9 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9857d1b8568dcee9"} 16:41:53 [16:41:53.631] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:df031e065ed89fdb epochNumber=1 {"provingJobId":"1:BASE_PARITY:df031e065ed89fdb"} 16:41:53 [16:41:53.632] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.632] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:23c885234082f907 epochNumber=1 {"provingJobId":"1:BASE_PARITY:23c885234082f907"} 16:41:53 [16:41:53.633] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:23b635e1c06e5de3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:23b635e1c06e5de3"} 16:41:53 [16:41:53.633] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8b3fbaaf0702d05c epochNumber=2 {"provingJobId":"1:BASE_PARITY:8b3fbaaf0702d05c"} 16:41:53 [16:41:53.633] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c77873f21d36824a epochNumber=1 {"provingJobId":"1:BASE_PARITY:c77873f21d36824a"} 16:41:53 [16:41:53.634] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.635] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5a3870bbe9131ea1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5a3870bbe9131ea1"} 16:41:53 [16:41:53.635] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6a3e1784e77fd365 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6a3e1784e77fd365"} 16:41:53 [16:41:53.636] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5a3870bbe9131ea1 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5a3870bbe9131ea1"} 16:41:53 [16:41:53.637] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.637] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b23512d0f7408233 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b23512d0f7408233"} 16:41:53 [16:41:53.638] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:166bb04d8afd2af7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:166bb04d8afd2af7"} 16:41:53 [16:41:53.638] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b23512d0f7408233 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:b23512d0f7408233"} 16:41:53 [16:41:53.639] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.639] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c08c8394ddc0f5d2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c08c8394ddc0f5d2"} 16:41:53 [16:41:53.640] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2c9957b9b28c5c43 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2c9957b9b28c5c43"} 16:41:53 [16:41:53.640] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c08c8394ddc0f5d2 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:c08c8394ddc0f5d2"} 16:41:53 [16:41:53.641] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.642] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c876197941ff37b5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c876197941ff37b5"} 16:41:53 [16:41:53.642] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c876197941ff37b5 {"provingJobId":"1:BASE_PARITY:c876197941ff37b5"} 16:41:53 [16:41:53.642] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c876197941ff37b5 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:c876197941ff37b5"} 16:41:53 [16:41:53.643] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:82e60301910dad79 epochNumber=1 {"provingJobId":"1:BASE_PARITY:82e60301910dad79"} 16:41:53 [16:41:53.643] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c876197941ff37b5 has already been completed {"provingJobId":"1:BASE_PARITY:c876197941ff37b5"} 16:41:53 [16:41:53.644] INFO: prover-client:proving-broker Proving Broker started 16:41:53 [16:41:53.645] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cbab3ccd004224ab epochNumber=1 {"provingJobId":"1:BASE_PARITY:cbab3ccd004224ab"} 16:41:53 [16:41:53.645] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:453dbb0a0aef1c86 epochNumber=2 {"provingJobId":"1:BASE_PARITY:453dbb0a0aef1c86"} 16:41:54 [16:41:54.699] INFO: prover-client:proving-broker Proving Broker started 16:41:54 [16:41:54.700] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cbab3ccd004224ab settled=false {"provingJobId":"1:BASE_PARITY:cbab3ccd004224ab","status":"pending"} 16:41:54 [16:41:54.701] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:453dbb0a0aef1c86 settled=false {"provingJobId":"1:BASE_PARITY:453dbb0a0aef1c86","status":"pending"} 16:41:54 [16:41:54.702] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:cbab3ccd004224ab type=BASE_PARITY startedAt=1750092113645 older agent has taken job {"provingJobId":"1:BASE_PARITY:cbab3ccd004224ab"} 16:41:54 [16:41:54.702] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cbab3ccd004224ab type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:cbab3ccd004224ab"} 16:41:54 [16:41:54.707] INFO: prover-client:proving-broker Proving Broker started 16:41:54 [16:41:54.708] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fe194123a985386a epochNumber=1 {"provingJobId":"1:BASE_PARITY:fe194123a985386a"} 16:41:54 [16:41:54.709] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1cd7d5145c8b665d epochNumber=2 {"provingJobId":"1:BASE_PARITY:1cd7d5145c8b665d"} 16:41:55 [16:41:55.710] INFO: prover-client:proving-broker Proving Broker started 16:41:55 [16:41:55.710] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fe194123a985386a settled=false {"provingJobId":"1:BASE_PARITY:fe194123a985386a","status":"pending"} 16:41:55 [16:41:55.711] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1cd7d5145c8b665d settled=false {"provingJobId":"1:BASE_PARITY:1cd7d5145c8b665d","status":"pending"} 16:41:55 [16:41:55.712] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:fe194123a985386a type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:fe194123a985386a"} 16:41:55 [16:41:55.713] INFO: prover-client:proving-broker Proving Broker started 16:41:55 [16:41:55.714] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d7e43c54d5c59c92 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d7e43c54d5c59c92"} 16:41:55 [16:41:55.714] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8cd0e548291e0804 epochNumber=2 {"provingJobId":"1:BASE_PARITY:8cd0e548291e0804"} 16:41:56 [16:41:56.717] INFO: prover-client:proving-broker Proving Broker started 16:41:56 [16:41:56.717] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d7e43c54d5c59c92 settled=false {"provingJobId":"1:BASE_PARITY:d7e43c54d5c59c92","status":"pending"} 16:41:56 [16:41:56.718] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8cd0e548291e0804 settled=false {"provingJobId":"1:BASE_PARITY:8cd0e548291e0804","status":"pending"} 16:41:56 [16:41:56.718] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d7e43c54d5c59c92 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d7e43c54d5c59c92"} 16:41:56 [16:41:56.718] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d7e43c54d5c59c92 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d7e43c54d5c59c92"} 16:41:56 [16:41:56.720] INFO: prover-client:proving-broker Proving Broker started 16:41:56 [16:41:56.721] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c420f3013fbcd5f9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c420f3013fbcd5f9"} 16:41:56 [16:41:56.721] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:385e12e3f6f91eb4 epochNumber=2 {"provingJobId":"1:BASE_PARITY:385e12e3f6f91eb4"} 16:41:56 [16:41:56.722] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c420f3013fbcd5f9 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:c420f3013fbcd5f9"} 16:41:56 [16:41:56.723] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:385e12e3f6f91eb4 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:385e12e3f6f91eb4"} 16:41:56 [16:41:56.724] INFO: prover-client:proving-broker Proving Broker started 16:41:56 [16:41:56.725] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:50bd7c72c0aff2c1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:50bd7c72c0aff2c1"} 16:41:56 [16:41:56.725] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:449b400b84b1787d epochNumber=2 {"provingJobId":"1:BASE_PARITY:449b400b84b1787d"} 16:41:56 [16:41:56.725] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:50bd7c72c0aff2c1 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:50bd7c72c0aff2c1"} 16:41:56 [16:41:56.726] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:50bd7c72c0aff2c1 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:50bd7c72c0aff2c1"} 16:41:56 [16:41:56.726] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:449b400b84b1787d type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:449b400b84b1787d"} 16:41:56 [16:41:56.727] INFO: prover-client:proving-broker Proving Broker started 16:41:56 [16:41:56.728] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:5647b99b7d61294d err=test error {"provingJoId":"1:BASE_PARITY:5647b99b7d61294d"} 16:41:56 [16:41:56.729] INFO: prover-client:proving-broker Proving Broker started 16:41:56 [16:41:56.729] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:fd68a56a4c863cca not found {"provingJobId":"1:BASE_PARITY:fd68a56a4c863cca"} 16:41:56 [16:41:56.731] INFO: prover-client:proving-broker Proving Broker started 16:41:56 [16:41:56.731] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f23f5af714afb6cc epochNumber=1 {"provingJobId":"1:BASE_PARITY:f23f5af714afb6cc"} 16:41:56 [16:41:56.732] INFO: prover-client:proving-broker Proving Broker started 16:41:56 [16:41:56.733] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:21d81f71e98cb0c9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:21d81f71e98cb0c9"} 16:41:56 [16:41:56.835] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:21d81f71e98cb0c9 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:21d81f71e98cb0c9"} 16:41:56 [16:41:56.842] INFO: prover-client:proving-broker Proving Broker started 16:41:56 [16:41:56.843] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b47b64d685ba686f epochNumber=1 {"provingJobId":"1:BASE_PARITY:b47b64d685ba686f"} 16:41:56 [16:41:56.949] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b47b64d685ba686f timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:b47b64d685ba686f"} 16:41:56 [16:41:56.954] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dc8bb5b93ca7b4f4 epochNumber=10 {"provingJobId":"1:BASE_PARITY:dc8bb5b93ca7b4f4"} 16:41:56 [16:41:56.975] INFO: prover-client:proving-broker Cleaned up jobs=1 16:41:57 [16:41:57.083] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.084] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:64ed33a523c3c395 epochNumber=1 {"provingJobId":"1:BASE_PARITY:64ed33a523c3c395"} 16:41:57 [16:41:57.260] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:64ed33a523c3c395 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:64ed33a523c3c395"} 16:41:57 [16:41:57.293] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.293] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7f18107a82f2cffd epochNumber=1 {"provingJobId":"1:BASE_PARITY:7f18107a82f2cffd"} 16:41:57 [16:41:57.294] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:7f18107a82f2cffd type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:7f18107a82f2cffd"} 16:41:57 [16:41:57.295] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.296] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1a2c0dbafe9d5c41 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1a2c0dbafe9d5c41"} 16:41:57 [16:41:57.296] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:1a2c0dbafe9d5c41 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:1a2c0dbafe9d5c41"} 16:41:57 [16:41:57.297] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1a2c0dbafe9d5c41 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:1a2c0dbafe9d5c41"} 16:41:57 [16:41:57.302] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.303] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ca7728e12948aa10 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ca7728e12948aa10"} 16:41:57 [16:41:57.304] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ca7728e12948aa10 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ca7728e12948aa10"} 16:41:57 [16:41:57.305] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.310] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4ec8a9458257ad4b epochNumber=1 {"provingJobId":"1:BASE_PARITY:4ec8a9458257ad4b"} 16:41:57 [16:41:57.311] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4ec8a9458257ad4b type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:4ec8a9458257ad4b"} 16:41:57 [16:41:57.312] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0079e725128bd3d9 epochNumber=3 {"provingJobId":"1:BASE_PARITY:0079e725128bd3d9"} 16:41:57 [16:41:57.335] INFO: prover-client:proving-broker Cleaned up jobs=1 16:41:57 [16:41:57.338] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4ec8a9458257ad4b err=test error {"provingJoId":"1:BASE_PARITY:4ec8a9458257ad4b"} 16:41:57 [16:41:57.340] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.340] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:727c69fc2800b362 settled=false {"provingJobId":"1:BASE_PARITY:727c69fc2800b362","status":"pending"} 16:41:57 [16:41:57.340] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9b9fca19ad8815f1 settled=false {"provingJobId":"1:BASE_PARITY:9b9fca19ad8815f1","status":"pending"} 16:41:57 [16:41:57.343] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.343] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:328b3271fc6b46d2 settled=true {"provingJobId":"1:BASE_PARITY:328b3271fc6b46d2","status":"fulfilled"} 16:41:57 [16:41:57.343] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:1e9ca410e3b2d771 settled=true {"provingJobId":"2:BASE_PARITY:1e9ca410e3b2d771","status":"fulfilled"} 16:41:57 [16:41:57.344] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.344] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9eaabfe8e1094667 settled=true {"provingJobId":"1:BASE_PARITY:9eaabfe8e1094667","status":"fulfilled"} 16:41:57 [16:41:57.345] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ce8d59567b00c402 settled=false {"provingJobId":"1:BASE_PARITY:ce8d59567b00c402","status":"pending"} 16:41:57 [16:41:57.346] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.347] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d79a3e5119bf39b7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d79a3e5119bf39b7"} 16:41:57 [16:41:57.348] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.349] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:05b6ce649d61e189 epochNumber=1 {"provingJobId":"1:BASE_PARITY:05b6ce649d61e189"} 16:41:57 [16:41:57.350] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:05b6ce649d61e189: Error: db error: Error: db error 16:41:57 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:05b6ce649d61e189"} 16:41:57 [16:41:57.353] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.354] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cce395fd299a5ba1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cce395fd299a5ba1"} 16:41:57 [16:41:57.354] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cce395fd299a5ba1 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cce395fd299a5ba1"} 16:41:57 [16:41:57.355] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cce395fd299a5ba1 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cce395fd299a5ba1"} 16:41:57 [16:41:57.357] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.357] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bf06624de36a5cc0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bf06624de36a5cc0"} 16:41:57 [16:41:57.358] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bf06624de36a5cc0 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:bf06624de36a5cc0"} 16:41:57 [16:41:57.358] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bf06624de36a5cc0 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:bf06624de36a5cc0"} 16:41:57 [16:41:57.358] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:bf06624de36a5cc0: Error: db error 16:41:57 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:bf06624de36a5cc0"} 16:41:57 [16:41:57.360] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.361] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9a5e644fa6e6b494 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9a5e644fa6e6b494"} 16:41:57 [16:41:57.361] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9a5e644fa6e6b494 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9a5e644fa6e6b494"} 16:41:57 [16:41:57.361] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9a5e644fa6e6b494 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9a5e644fa6e6b494"} 16:41:57 [16:41:57.362] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.363] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b2e9864257b1855c epochNumber=1 {"provingJobId":"1:BASE_PARITY:b2e9864257b1855c"} 16:41:57 [16:41:57.364] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b2e9864257b1855c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b2e9864257b1855c"} 16:41:57 [16:41:57.364] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b2e9864257b1855c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b2e9864257b1855c"} 16:41:57 [16:41:57.364] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:b2e9864257b1855c jobErr=test error: Error: db error 16:41:57 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:b2e9864257b1855c"} 16:41:57 [16:41:57.365] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.366] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ea73d01d399f048d not found {"provingJobId":"1:BASE_PARITY:ea73d01d399f048d"} 16:41:57 [16:41:57.366] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.367] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0f4a1ce72c258c96 err=test error {"provingJoId":"1:BASE_PARITY:0f4a1ce72c258c96"} 16:41:57 [16:41:57.369] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.380] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:458693aad7cf520a epochNumber=1 {"provingJobId":"1:BASE_PARITY:458693aad7cf520a"} 16:41:57 [16:41:57.381] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:458693aad7cf520a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:458693aad7cf520a"} 16:41:57 [16:41:57.381] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:458693aad7cf520a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:458693aad7cf520a"} 16:41:57 [16:41:57.394] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:0bab51a3b049f688 epochNumber=2 {"provingJobId":"2:BASE_PARITY:0bab51a3b049f688"} 16:41:57 [16:41:57.395] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:0bab51a3b049f688 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:0bab51a3b049f688"} 16:41:57 [16:41:57.395] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:0bab51a3b049f688 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:0bab51a3b049f688"} 16:41:57 [16:41:57.456] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:f37897f68dc0d9ae epochNumber=3 {"provingJobId":"3:BASE_PARITY:f37897f68dc0d9ae"} 16:41:57 [16:41:57.474] INFO: prover-client:proving-broker Cleaned up jobs=1 16:41:57 [16:41:57.517] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:5aaa6481d5f07f06 epochNumber=4 {"provingJobId":"4:BASE_PARITY:5aaa6481d5f07f06"} 16:41:57 [16:41:57.525] INFO: prover-client:proving-broker Cleaned up jobs=1 16:41:57 [16:41:57.579] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:57f08e58a0e55aae epochNumber=5 {"provingJobId":"5:BASE_PARITY:57f08e58a0e55aae"} 16:41:57 [16:41:57.602] INFO: prover-client:proving-broker Cleaned up jobs=1 16:41:57 [16:41:57.633] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.634] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c434507e40732114 epochNumber=42 {"provingJobId":"1:BASE_PARITY:c434507e40732114"} 16:41:57 [16:41:57.635] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testA8D1cr/42 with map size 1073741824 16:41:57 [16:41:57.639] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.668] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:3898f1c94532c078 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3898f1c94532c078"} 16:41:57 [16:41:57.674] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.675] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:61b1aaaea192a02c epochNumber=1 {"provingJobId":"1:BASE_PARITY:61b1aaaea192a02c"} 16:41:57 [16:41:57.676] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcKRQPz/1 with map size 1073741824 16:41:57 [16:41:57.679] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.708] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:adfdb2044784cd9d epochNumber=1 {"provingJobId":"1:BASE_PARITY:adfdb2044784cd9d"} 16:41:57 [16:41:57.716] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.717] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:560b7f6998b8b163 epochNumber=1 {"provingJobId":"1:BASE_PARITY:560b7f6998b8b163"} 16:41:57 [16:41:57.718] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUjLHSP/1 with map size 1073741824 16:41:57 [16:41:57.720] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.741] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:560b7f6998b8b163 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:560b7f6998b8b163"} 16:41:57 [16:41:57.746] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.746] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:53673971656f1345 epochNumber=1 {"provingJobId":"1:BASE_PARITY:53673971656f1345"} 16:41:57 [16:41:57.747] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHwdcyB/1 with map size 1073741824 16:41:57 [16:41:57.749] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.769] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:53673971656f1345 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:53673971656f1345"} 16:41:57 [16:41:57.769] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:53673971656f1345 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:53673971656f1345"} 16:41:57 [16:41:57.777] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:53673971656f1345 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:53673971656f1345"} 16:41:57 [16:41:57.781] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.782] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:379ae74fde2790f4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:379ae74fde2790f4"} 16:41:57 [16:41:57.783] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testY4iILj/1 with map size 1073741824 16:41:57 [16:41:57.785] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.814] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.817] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.818] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:32b29dda015082f3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:32b29dda015082f3"} 16:41:57 [16:41:57.818] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJAeL47/1 with map size 1073741824 16:41:57 [16:41:57.821] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.841] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:32b29dda015082f3 {"provingJobId":"1:BASE_PARITY:32b29dda015082f3"} 16:41:57 [16:41:57.842] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:32b29dda015082f3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:32b29dda015082f3"} 16:41:57 [16:41:57.842] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:32b29dda015082f3 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:32b29dda015082f3"} 16:41:57 [16:41:57.853] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.854] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e99a846ddb0f436e epochNumber=1 {"provingJobId":"1:BASE_PARITY:e99a846ddb0f436e"} 16:41:57 [16:41:57.855] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testK0mV0p/1 with map size 1073741824 16:41:57 [16:41:57.858] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.873] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e99a846ddb0f436e {"provingJobId":"1:BASE_PARITY:e99a846ddb0f436e"} 16:41:57 [16:41:57.873] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e99a846ddb0f436e type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:e99a846ddb0f436e"} 16:41:57 [16:41:57.883] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.883] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:56c5f91e9c244f91 epochNumber=1 {"provingJobId":"1:BASE_PARITY:56c5f91e9c244f91"} 16:41:57 [16:41:57.884] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaizSFw/1 with map size 1073741824 16:41:57 [16:41:57.886] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.903] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:56c5f91e9c244f91 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:56c5f91e9c244f91"} 16:41:57 [16:41:57.903] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:56c5f91e9c244f91 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:56c5f91e9c244f91"} 16:41:57 [16:41:57.911] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.912] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1efc0ccf08c71468 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1efc0ccf08c71468"} 16:41:57 [16:41:57.913] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUkt2rc/1 with map size 1073741824 16:41:57 [16:41:57.916] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.932] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1efc0ccf08c71468 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:1efc0ccf08c71468"} 16:41:57 [16:41:57.933] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1efc0ccf08c71468 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:1efc0ccf08c71468"} 16:41:57 [16:41:57.941] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.942] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a11051f5b447052a epochNumber=0 {"provingJobId":"1:BASE_PARITY:a11051f5b447052a"} 16:41:57 [16:41:57.943] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testCvxebo/0 with map size 1073741824 16:41:57 [16:41:57.946] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:57 [16:41:57.970] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a11051f5b447052a epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:a11051f5b447052a"} 16:41:57 [16:41:57.971] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2fef849588c0e0dc epochNumber=0 {"provingJobId":"1:BASE_PARITY:2fef849588c0e0dc"} 16:41:57 [16:41:57.972] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a11051f5b447052a epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:a11051f5b447052a"} 16:41:57 [16:41:57.973] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e7c5c816b00d520c epochNumber=0 {"provingJobId":"1:BASE_PARITY:e7c5c816b00d520c"} 16:41:57 [16:41:57.991] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.996] INFO: prover-client:proving-broker Proving Broker started 16:41:57 [16:41:57.997] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dd22d8be2c11df48 epochNumber=2 {"provingJobId":"1:BASE_PARITY:dd22d8be2c11df48"} 16:41:57 [16:41:57.998] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTArI0P/2 with map size 1073741824 16:41:58 [16:41:58.000] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.017] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:906024115dfb5c66 epochNumber=1 {"provingJobId":"1:BASE_PARITY:906024115dfb5c66"} 16:41:58 [16:41:58.019] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTArI0P/1 with map size 1073741824 16:41:58 [16:41:58.021] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.041] INFO: prover-client:proving-broker Proving Broker started 16:41:58 [16:41:58.041] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e38a996a3aa63c91 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e38a996a3aa63c91"} 16:41:58 [16:41:58.042] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2g2uRp/1 with map size 1073741824 16:41:58 [16:41:58.043] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.062] INFO: prover-client:proving-broker Proving Broker started 16:41:58 [16:41:58.063] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:41904de2909fef6b epochNumber=1 {"provingJobId":"1:BASE_PARITY:41904de2909fef6b"} 16:41:58 [16:41:58.064] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYhmbIa/1 with map size 1073741824 16:41:58 [16:41:58.067] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.085] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:78f69b3757c8676b epochNumber=1 {"provingJobId":"1:BASE_PARITY:78f69b3757c8676b"} 16:41:58 [16:41:58.094] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e2b849f1e9953f32 epochNumber=2 {"provingJobId":"1:BASE_PARITY:e2b849f1e9953f32"} 16:41:58 [16:41:58.096] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testYhmbIa/2 with map size 1073741824 16:41:58 [16:41:58.102] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.122] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f2d78973d2cff3b2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f2d78973d2cff3b2"} 16:41:58 [16:41:58.132] INFO: prover-client:proving-broker Proving Broker started 16:41:58 [16:41:58.133] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:64f089080b43f168 epochNumber=1 {"provingJobId":"1:BASE_PARITY:64f089080b43f168"} 16:41:58 [16:41:58.134] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testS9VnDB/1 with map size 1073741824 16:41:58 [16:41:58.136] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.155] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6dd08e873df35149 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6dd08e873df35149"} 16:41:58 [16:41:58.162] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a24d4b2ef1c8142d epochNumber=2 {"provingJobId":"1:BASE_PARITY:a24d4b2ef1c8142d"} 16:41:58 [16:41:58.163] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testS9VnDB/2 with map size 1073741824 16:41:58 [16:41:58.166] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.184] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2b92c8438f15c11f epochNumber=1 {"provingJobId":"1:BASE_PARITY:2b92c8438f15c11f"} 16:41:58 [16:41:58.213] INFO: prover-client:proving-broker Proving Broker started 16:41:58 [16:41:58.214] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03b16e27abb379c1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:03b16e27abb379c1"} 16:41:58 [16:41:58.215] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7cUPuD/1 with map size 1073741824 16:41:58 [16:41:58.217] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.235] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1be48282c99bf516 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1be48282c99bf516"} 16:41:58 [16:41:58.241] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2d776b94268542a5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:2d776b94268542a5"} 16:41:58 [16:41:58.242] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7cUPuD/2 with map size 1073741824 16:41:58 [16:41:58.244] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.260] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:35fc4cc5fa2d20ad epochNumber=1 {"provingJobId":"1:BASE_PARITY:35fc4cc5fa2d20ad"} 16:41:58 [16:41:58.270] INFO: prover-client:proving-broker Proving Broker started 16:41:58 [16:41:58.271] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cba39fa9f200af81 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cba39fa9f200af81"} 16:41:58 [16:41:58.271] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9fqhms/1 with map size 1073741824 16:41:58 [16:41:58.273] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.293] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4ec17d43ae58a9a2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4ec17d43ae58a9a2"} 16:41:58 [16:41:58.301] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cba39fa9f200af81 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cba39fa9f200af81"} 16:41:58 [16:41:58.313] INFO: prover-client:proving-broker Proving Broker started 16:41:58 [16:41:58.314] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4c4f4ea6640b4d66 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4c4f4ea6640b4d66"} 16:41:58 [16:41:58.315] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testu68ztm/1 with map size 1073741824 16:41:58 [16:41:58.317] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.333] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5ae093df458a0541 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5ae093df458a0541"} 16:41:58 [16:41:58.339] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4c4f4ea6640b4d66 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:4c4f4ea6640b4d66"} 16:41:58 [16:41:58.349] INFO: prover-client:proving-broker Proving Broker started 16:41:58 [16:41:58.349] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eae6b91dfc836668 epochNumber=1 {"provingJobId":"1:BASE_PARITY:eae6b91dfc836668"} 16:41:58 [16:41:58.350] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCDWfDr/1 with map size 1073741824 16:41:58 [16:41:58.352] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.369] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d0c97d9f0ee00d62 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d0c97d9f0ee00d62"} 16:41:58 [16:41:58.375] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:eae6b91dfc836668 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:eae6b91dfc836668"} 16:41:58 [16:41:58.379] INFO: prover-client:proving-broker Proving Broker started 16:41:58 [16:41:58.380] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6b7fb7bdee552645 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6b7fb7bdee552645"} 16:41:58 [16:41:58.381] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0DSfxK/1 with map size 1073741824 16:41:58 [16:41:58.384] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.409] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6b7fb7bdee552645 {"provingJobId":"1:BASE_PARITY:6b7fb7bdee552645"} 16:41:58 [16:41:58.409] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6b7fb7bdee552645 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:6b7fb7bdee552645"} 16:41:58 [16:41:58.417] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:36b6e7c357f8d608 epochNumber=1 {"provingJobId":"1:BASE_PARITY:36b6e7c357f8d608"} 16:41:58 [16:41:58.424] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6b7fb7bdee552645 has already been completed {"provingJobId":"1:BASE_PARITY:6b7fb7bdee552645"} 16:41:58 [16:41:58.428] INFO: prover-client:proving-broker Proving Broker started 16:41:58 [16:41:58.428] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:07c4c7d7f42f9fa6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:07c4c7d7f42f9fa6"} 16:41:58 [16:41:58.429] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjP2U9H/1 with map size 1073741824 16:41:58 [16:41:58.431] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:58 [16:41:58.448] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6bf3cefc6f34be80 epochNumber=2 {"provingJobId":"1:BASE_PARITY:6bf3cefc6f34be80"} 16:41:58 [16:41:58.449] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testjP2U9H/2 with map size 1073741824 16:41:58 [16:41:58.451] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:59 [16:41:59.520] INFO: prover-client:proving-broker Proving Broker started 16:41:59 [16:41:59.527] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:07c4c7d7f42f9fa6 settled=false {"provingJobId":"1:BASE_PARITY:07c4c7d7f42f9fa6","status":"pending"} 16:41:59 [16:41:59.529] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6bf3cefc6f34be80 settled=false {"provingJobId":"1:BASE_PARITY:6bf3cefc6f34be80","status":"pending"} 16:41:59 [16:41:59.531] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:07c4c7d7f42f9fa6 type=BASE_PARITY startedAt=1750092118468 older agent has taken job {"provingJobId":"1:BASE_PARITY:07c4c7d7f42f9fa6"} 16:41:59 [16:41:59.531] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:07c4c7d7f42f9fa6 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:07c4c7d7f42f9fa6"} 16:41:59 [16:41:59.537] INFO: prover-client:proving-broker Proving Broker started 16:41:59 [16:41:59.538] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e7aaa030006b6a4c epochNumber=1 {"provingJobId":"1:BASE_PARITY:e7aaa030006b6a4c"} 16:41:59 [16:41:59.539] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4mWQBU/1 with map size 1073741824 16:41:59 [16:41:59.541] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:41:59 [16:41:59.561] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1742ba8e0f7227fd epochNumber=2 {"provingJobId":"1:BASE_PARITY:1742ba8e0f7227fd"} 16:41:59 [16:41:59.562] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test4mWQBU/2 with map size 1073741824 16:41:59 [16:41:59.564] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:00 [16:42:00.586] INFO: prover-client:proving-broker Proving Broker started 16:42:00 [16:42:00.590] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e7aaa030006b6a4c settled=false {"provingJobId":"1:BASE_PARITY:e7aaa030006b6a4c","status":"pending"} 16:42:00 [16:42:00.592] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1742ba8e0f7227fd settled=false {"provingJobId":"1:BASE_PARITY:1742ba8e0f7227fd","status":"pending"} 16:42:00 [16:42:00.593] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e7aaa030006b6a4c type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:e7aaa030006b6a4c"} 16:42:00 [16:42:00.598] INFO: prover-client:proving-broker Proving Broker started 16:42:00 [16:42:00.599] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c946c66e3de650fd epochNumber=1 {"provingJobId":"1:BASE_PARITY:c946c66e3de650fd"} 16:42:00 [16:42:00.600] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFA0mgF/1 with map size 1073741824 16:42:00 [16:42:00.609] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:00 [16:42:00.629] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:56b782c82071d881 epochNumber=2 {"provingJobId":"1:BASE_PARITY:56b782c82071d881"} 16:42:00 [16:42:00.630] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFA0mgF/2 with map size 1073741824 16:42:00 [16:42:00.633] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:01 [16:42:01.653] INFO: prover-client:proving-broker Proving Broker started 16:42:01 [16:42:01.655] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c946c66e3de650fd settled=false {"provingJobId":"1:BASE_PARITY:c946c66e3de650fd","status":"pending"} 16:42:01 [16:42:01.657] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:56b782c82071d881 settled=false {"provingJobId":"1:BASE_PARITY:56b782c82071d881","status":"pending"} 16:42:01 [16:42:01.659] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c946c66e3de650fd type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c946c66e3de650fd"} 16:42:01 [16:42:01.660] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c946c66e3de650fd type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:c946c66e3de650fd"} 16:42:01 [16:42:01.673] INFO: prover-client:proving-broker Proving Broker started 16:42:01 [16:42:01.674] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:06c483588b0cc095 epochNumber=1 {"provingJobId":"1:BASE_PARITY:06c483588b0cc095"} 16:42:01 [16:42:01.677] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRG5457/1 with map size 1073741824 16:42:01 [16:42:01.680] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:01 [16:42:01.707] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a8abbc8c485ff9e9 epochNumber=2 {"provingJobId":"1:BASE_PARITY:a8abbc8c485ff9e9"} 16:42:01 [16:42:01.708] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testRG5457/2 with map size 1073741824 16:42:01 [16:42:01.711] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:01 [16:42:01.735] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:06c483588b0cc095 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:06c483588b0cc095"} 16:42:01 [16:42:01.741] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a8abbc8c485ff9e9 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:a8abbc8c485ff9e9"} 16:42:01 [16:42:01.753] INFO: prover-client:proving-broker Proving Broker started 16:42:01 [16:42:01.754] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f12b2fef46504c8f epochNumber=1 {"provingJobId":"1:BASE_PARITY:f12b2fef46504c8f"} 16:42:01 [16:42:01.755] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testoHiCYe/1 with map size 1073741824 16:42:01 [16:42:01.758] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:01 [16:42:01.780] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2230e80f674aa982 epochNumber=2 {"provingJobId":"1:BASE_PARITY:2230e80f674aa982"} 16:42:01 [16:42:01.782] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testoHiCYe/2 with map size 1073741824 16:42:01 [16:42:01.783] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:01 [16:42:01.810] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f12b2fef46504c8f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f12b2fef46504c8f"} 16:42:01 [16:42:01.811] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f12b2fef46504c8f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f12b2fef46504c8f"} 16:42:01 [16:42:01.816] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2230e80f674aa982 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:2230e80f674aa982"} 16:42:01 [16:42:01.826] INFO: prover-client:proving-broker Proving Broker started 16:42:01 [16:42:01.826] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:eee070fe69c16825 err=test error {"provingJoId":"1:BASE_PARITY:eee070fe69c16825"} 16:42:01 [16:42:01.829] INFO: prover-client:proving-broker Proving Broker started 16:42:01 [16:42:01.829] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:05ef427084941a5c not found {"provingJobId":"1:BASE_PARITY:05ef427084941a5c"} 16:42:01 [16:42:01.834] INFO: prover-client:proving-broker Proving Broker started 16:42:01 [16:42:01.835] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5fe5f53b0f7c4003 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5fe5f53b0f7c4003"} 16:42:01 [16:42:01.837] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testymzqdH/1 with map size 1073741824 16:42:01 [16:42:01.841] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:01 [16:42:01.861] INFO: prover-client:proving-broker Proving Broker started 16:42:01 [16:42:01.863] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:89a065e21b2c8da1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:89a065e21b2c8da1"} 16:42:01 [16:42:01.864] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKeodLl/1 with map size 1073741824 16:42:01 [16:42:01.868] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:01 [16:42:01.988] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:89a065e21b2c8da1 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:89a065e21b2c8da1"} 16:42:01 [16:42:01.997] INFO: prover-client:proving-broker Proving Broker started 16:42:01 [16:42:01.998] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c16f7653779cf1c7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c16f7653779cf1c7"} 16:42:01 [16:42:02.000] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVEBAHr/1 with map size 1073741824 16:42:02 [16:42:02.002] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.127] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c16f7653779cf1c7 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:c16f7653779cf1c7"} 16:42:02 [16:42:02.129] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b81aa81d168af860 epochNumber=10 {"provingJobId":"1:BASE_PARITY:b81aa81d168af860"} 16:42:02 [16:42:02.130] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testVEBAHr/10 with map size 1073741824 16:42:02 [16:42:02.132] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.153] INFO: prover-client:proving-broker Cleaned up jobs=1 16:42:02 [16:42:02.153] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:42:02 [16:42:02.278] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.279] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:505aaf3488a37f47 epochNumber=1 {"provingJobId":"1:BASE_PARITY:505aaf3488a37f47"} 16:42:02 [16:42:02.281] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpxUDdQ/1 with map size 1073741824 16:42:02 [16:42:02.283] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.481] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:505aaf3488a37f47 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:505aaf3488a37f47"} 16:42:02 [16:42:02.515] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.516] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0990128214208429 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0990128214208429"} 16:42:02 [16:42:02.516] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWGGlSv/1 with map size 1073741824 16:42:02 [16:42:02.522] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.543] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0990128214208429 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:0990128214208429"} 16:42:02 [16:42:02.547] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.548] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ec8d0f9474fe2474 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ec8d0f9474fe2474"} 16:42:02 [16:42:02.549] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPA5jhk/1 with map size 1073741824 16:42:02 [16:42:02.551] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.571] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:ec8d0f9474fe2474 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:ec8d0f9474fe2474"} 16:42:02 [16:42:02.572] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ec8d0f9474fe2474 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:ec8d0f9474fe2474"} 16:42:02 [16:42:02.584] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.585] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7bd1227893d4283e epochNumber=1 {"provingJobId":"1:BASE_PARITY:7bd1227893d4283e"} 16:42:02 [16:42:02.587] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbFROTZ/1 with map size 1073741824 16:42:02 [16:42:02.595] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.618] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7bd1227893d4283e type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:7bd1227893d4283e"} 16:42:02 [16:42:02.629] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.630] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c42cf81f286d1988 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c42cf81f286d1988"} 16:42:02 [16:42:02.631] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testE8ri29/1 with map size 1073741824 16:42:02 [16:42:02.635] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.655] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c42cf81f286d1988 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:c42cf81f286d1988"} 16:42:02 [16:42:02.656] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2529f4101af0a561 epochNumber=3 {"provingJobId":"1:BASE_PARITY:2529f4101af0a561"} 16:42:02 [16:42:02.658] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testE8ri29/3 with map size 1073741824 16:42:02 [16:42:02.662] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.682] INFO: prover-client:proving-broker Cleaned up jobs=1 16:42:02 [16:42:02.683] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:42:02 [16:42:02.706] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c42cf81f286d1988 err=test error {"provingJoId":"1:BASE_PARITY:c42cf81f286d1988"} 16:42:02 [16:42:02.715] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testC1L33s/1 with map size 1073741824 16:42:02 [16:42:02.718] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.742] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testC1L33s/2 with map size 1073741824 16:42:02 [16:42:02.746] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.769] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.772] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0a1c133ad22ef11f settled=false {"provingJobId":"1:BASE_PARITY:0a1c133ad22ef11f","status":"pending"} 16:42:02 [16:42:02.774] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:24ec4484ee1ddca4 settled=false {"provingJobId":"1:BASE_PARITY:24ec4484ee1ddca4","status":"pending"} 16:42:02 [16:42:02.784] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testuDLnjo/1 with map size 1073741824 16:42:02 [16:42:02.787] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.813] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testuDLnjo/2 with map size 1073741824 16:42:02 [16:42:02.818] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.852] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.854] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f11764d4bccf27f9 settled=true {"provingJobId":"1:BASE_PARITY:f11764d4bccf27f9","status":"fulfilled"} 16:42:02 [16:42:02.857] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b624e4a94f281452 settled=true {"provingJobId":"2:BASE_PARITY:b624e4a94f281452","status":"fulfilled"} 16:42:02 [16:42:02.863] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqS1VDR/1 with map size 1073741824 16:42:02 [16:42:02.867] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.900] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testqS1VDR/2 with map size 1073741824 16:42:02 [16:42:02.905] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.927] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.928] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ffa1fe350e04d348 settled=true {"provingJobId":"1:BASE_PARITY:ffa1fe350e04d348","status":"fulfilled"} 16:42:02 [16:42:02.930] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:efa334194ba9f415 settled=false {"provingJobId":"1:BASE_PARITY:efa334194ba9f415","status":"pending"} 16:42:02 [16:42:02.937] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.937] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:164ff00b315313f9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:164ff00b315313f9"} 16:42:02 [16:42:02.940] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOUzG27/1 with map size 1073741824 16:42:02 [16:42:02.942] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:02 [16:42:02.968] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.968] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:382bb37bec3b67c5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:382bb37bec3b67c5"} 16:42:02 [16:42:02.969] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:382bb37bec3b67c5: Error: db error: Error: db error 16:42:02 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:382bb37bec3b67c5"} 16:42:02 [16:42:02.974] INFO: prover-client:proving-broker Proving Broker started 16:42:02 [16:42:02.975] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:76e57b87ab0113ec epochNumber=1 {"provingJobId":"1:BASE_PARITY:76e57b87ab0113ec"} 16:42:02 [16:42:02.976] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLEb88D/1 with map size 1073741824 16:42:02 [16:42:02.980] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:03 [16:42:03.004] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:76e57b87ab0113ec type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:76e57b87ab0113ec"} 16:42:03 [16:42:03.004] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:76e57b87ab0113ec type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:76e57b87ab0113ec"} 16:42:03 [16:42:03.021] INFO: prover-client:proving-broker Proving Broker started 16:42:03 [16:42:03.022] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7954cdd7ab0589f6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7954cdd7ab0589f6"} 16:42:03 [16:42:03.025] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9IJDcL/1 with map size 1073741824 16:42:03 [16:42:03.027] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:03 [16:42:03.050] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7954cdd7ab0589f6 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:7954cdd7ab0589f6"} 16:42:03 [16:42:03.050] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7954cdd7ab0589f6 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:7954cdd7ab0589f6"} 16:42:03 [16:42:03.051] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:7954cdd7ab0589f6: Error: db error 16:42:03 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:7954cdd7ab0589f6"} 16:42:03 [16:42:03.056] INFO: prover-client:proving-broker Proving Broker started 16:42:03 [16:42:03.056] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9fde892f05c44b34 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9fde892f05c44b34"} 16:42:03 [16:42:03.058] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testo1FoC1/1 with map size 1073741824 16:42:03 [16:42:03.059] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:03 [16:42:03.083] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9fde892f05c44b34 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9fde892f05c44b34"} 16:42:03 [16:42:03.084] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9fde892f05c44b34 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9fde892f05c44b34"} 16:42:03 [16:42:03.101] INFO: prover-client:proving-broker Proving Broker started 16:42:03 [16:42:03.101] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a89e171f22f75bf5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a89e171f22f75bf5"} 16:42:03 [16:42:03.102] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3ktijj/1 with map size 1073741824 16:42:03 [16:42:03.105] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:03 [16:42:03.123] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a89e171f22f75bf5 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a89e171f22f75bf5"} 16:42:03 [16:42:03.124] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a89e171f22f75bf5 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:a89e171f22f75bf5"} 16:42:03 [16:42:03.125] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a89e171f22f75bf5 jobErr=test error: Error: db error 16:42:03 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:a89e171f22f75bf5"} 16:42:03 [16:42:03.131] INFO: prover-client:proving-broker Proving Broker started 16:42:03 [16:42:03.131] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3c9b81f0c4003609 not found {"provingJobId":"1:BASE_PARITY:3c9b81f0c4003609"} 16:42:03 [16:42:03.135] INFO: prover-client:proving-broker Proving Broker started 16:42:03 [16:42:03.135] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:a7f54e37aa78ce74 err=test error {"provingJoId":"1:BASE_PARITY:a7f54e37aa78ce74"} 16:42:03 [16:42:03.140] INFO: prover-client:proving-broker Proving Broker started 16:42:03 [16:42:03.152] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f133e3b57da387e7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f133e3b57da387e7"} 16:42:03 [16:42:03.153] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLt8D7j/1 with map size 1073741824 16:42:03 [16:42:03.156] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:03 [16:42:03.179] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f133e3b57da387e7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f133e3b57da387e7"} 16:42:03 [16:42:03.180] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f133e3b57da387e7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f133e3b57da387e7"} 16:42:03 [16:42:03.195] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:15f846899b931cd7 epochNumber=2 {"provingJobId":"2:BASE_PARITY:15f846899b931cd7"} 16:42:03 [16:42:03.196] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testLt8D7j/2 with map size 1073741824 16:42:03 [16:42:03.201] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:03 [16:42:03.221] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:15f846899b931cd7 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:15f846899b931cd7"} 16:42:03 [16:42:03.222] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:15f846899b931cd7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:15f846899b931cd7"} 16:42:03 [16:42:03.292] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:fbe1d4411c0d03e9 epochNumber=3 {"provingJobId":"3:BASE_PARITY:fbe1d4411c0d03e9"} 16:42:03 [16:42:03.294] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testLt8D7j/3 with map size 1073741824 16:42:03 [16:42:03.297] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:03 [16:42:03.319] INFO: prover-client:proving-broker Cleaned up jobs=1 16:42:03 [16:42:03.320] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:42:03 [16:42:03.381] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:7a91bd48aa0e9c05 epochNumber=4 {"provingJobId":"4:BASE_PARITY:7a91bd48aa0e9c05"} 16:42:03 [16:42:03.385] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testLt8D7j/4 with map size 1073741824 16:42:03 [16:42:03.402] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:03 [16:42:03.451] INFO: prover-client:proving-broker Cleaned up jobs=1 16:42:03 [16:42:03.451] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 16:42:03 [16:42:03.497] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:3e9c0397d4bb5399 epochNumber=5 {"provingJobId":"5:BASE_PARITY:3e9c0397d4bb5399"} 16:42:03 [16:42:03.500] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testLt8D7j/5 with map size 1073741824 16:42:03 [16:42:03.503] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:42:03 [16:42:03.534] INFO: prover-client:proving-broker Cleaned up jobs=1 16:42:03 [16:42:03.535] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 16:42:03 PASS src/proving_broker/proving_broker.test.ts (13.501 s) 16:42:03 ProvingBroker 16:42:03 Producer API 16:42:03 ✓ refuses stale jobs (52 ms) 16:42:03 ✓ enqueues jobs (5 ms) 16:42:03 ✓ ignores duplicate jobs (4 ms) 16:42:03 ✓ reports correct status when enqueuing repeat jobs (4 ms) 16:42:03 ✓ throws an error in case of duplicate job IDs (37 ms) 16:42:03 ✓ returns not-found status for non-existing jobs (3 ms) 16:42:03 ✓ cancels jobs in queue (5 ms) 16:42:03 ✓ cancels jobs in-progress (4 ms) 16:42:03 ✓ returns job result if successful (5 ms) 16:42:03 ✓ returns job error if failed (4 ms) 16:42:03 ✓ correctly returns job status for concurrent writes (4 ms) 16:42:03 ✓ refuses stale jobs (40 ms) 16:42:03 ✓ enqueues jobs (43 ms) 16:42:03 ✓ ignores duplicate jobs (28 ms) 16:42:03 ✓ reports correct status when enqueuing repeat jobs (34 ms) 16:42:03 ✓ throws an error in case of duplicate job IDs (33 ms) 16:42:03 ✓ returns not-found status for non-existing jobs (2 ms) 16:42:03 ✓ cancels jobs in queue (34 ms) 16:42:03 ✓ cancels jobs in-progress (30 ms) 16:42:03 ✓ returns job result if successful (29 ms) 16:42:03 ✓ returns job error if failed (29 ms) 16:42:03 ✓ correctly returns job status for concurrent writes (47 ms) 16:42:03 Consumer API 16:42:03 ✓ returns undefined if no jobs are available (1 ms) 16:42:03 ✓ returns jobs in priority order (3 ms) 16:42:03 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 16:42:03 ✓ returns a job if it is in the allowList (3 ms) 16:42:03 ✓ returns the most important job if it is in the allowList (2 ms) 16:42:03 ✓ returns any job if filter is empty (3 ms) 16:42:03 ✓ returns a new job when reporting job success (2 ms) 16:42:03 ✓ returns a new job when reporting permanent error (2 ms) 16:42:03 ✓ returns a new job when reporting retry-able error (2 ms) 16:42:03 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 16:42:03 ✓ returns a new job if job is already in progress elsewhere (1059 ms) 16:42:03 ✓ avoids sending the same job to a new agent after a restart (1006 ms) 16:42:03 ✓ avoids sending a completed job to a new agent after a restart (1006 ms) 16:42:03 ✓ tracks job result if in progress (3 ms) 16:42:03 ✓ tracks job result even if job is in queue (3 ms) 16:42:03 ✓ ignores reported job error if unknown job (1 ms) 16:42:03 ✓ ignores job result if unknown job (1 ms) 16:42:03 ✓ returns undefined if no jobs are available (5 ms) 16:42:03 ✓ returns jobs in priority order (47 ms) 16:42:03 ✓ returns undefined if no jobs are available for the given allowList (20 ms) 16:42:03 ✓ returns a job if it is in the allowList (69 ms) 16:42:03 ✓ returns the most important job if it is in the allowList (79 ms) 16:42:03 ✓ returns any job if filter is empty (58 ms) 16:42:03 ✓ returns a new job when reporting job success (42 ms) 16:42:03 ✓ returns a new job when reporting permanent error (35 ms) 16:42:03 ✓ returns a new job when reporting retry-able error (30 ms) 16:42:03 ✓ returns a new job when reporting progress if current one is cancelled (49 ms) 16:42:03 ✓ returns a new job if job is already in progress elsewhere (1108 ms) 16:42:03 ✓ avoids sending the same job to a new agent after a restart (1062 ms) 16:42:03 ✓ avoids sending a completed job to a new agent after a restart (1074 ms) 16:42:03 ✓ tracks job result if in progress (81 ms) 16:42:03 ✓ tracks job result even if job is in queue (72 ms) 16:42:03 ✓ ignores reported job error if unknown job (3 ms) 16:42:03 ✓ ignores job result if unknown job (3 ms) 16:42:03 Timeouts 16:42:03 ✓ tracks in progress jobs (2 ms) 16:42:03 ✓ re-enqueues jobs that time out (110 ms) 16:42:03 ✓ cancel stale jobs that time out (240 ms) 16:42:03 ✓ keeps the jobs in progress while it is alive (208 ms) 16:42:03 ✓ tracks in progress jobs (28 ms) 16:42:03 ✓ re-enqueues jobs that time out (135 ms) 16:42:03 ✓ cancel stale jobs that time out (280 ms) 16:42:03 ✓ keeps the jobs in progress while it is alive (235 ms) 16:42:03 Retries 16:42:03 ✓ retries jobs (3 ms) 16:42:03 ✓ retries up to a maximum number of times (5 ms) 16:42:03 ✓ passing retry=false does not retry (3 ms) 16:42:03 ✓ does not retry if job is stale (34 ms) 16:42:03 ✓ retries jobs (33 ms) 16:42:03 ✓ retries up to a maximum number of times (37 ms) 16:42:03 ✓ passing retry=false does not retry (44 ms) 16:42:03 ✓ does not retry if job is stale (84 ms) 16:42:03 Database management 16:42:03 ✓ re-enqueues proof requests on start (3 ms) 16:42:03 ✓ restores proof results on start (2 ms) 16:42:03 ✓ only re-enqueues unfinished jobs (1 ms) 16:42:03 ✓ saves job when enqueued (2 ms) 16:42:03 ✓ does not retain job if database fails to save (4 ms) 16:42:03 ✓ saves job result (3 ms) 16:42:03 ✓ saves result even if database fails to save (3 ms) 16:42:03 ✓ saves job error (2 ms) 16:42:03 ✓ saves job error even if database fails to save (3 ms) 16:42:03 ✓ does not save job result if job is unknown (1 ms) 16:42:03 ✓ does not save job error if job is unknown (2 ms) 16:42:03 ✓ cleans up old jobs periodically (262 ms) 16:42:03 ✓ re-enqueues proof requests on start (69 ms) 16:42:03 ✓ restores proof results on start (79 ms) 16:42:03 ✓ only re-enqueues unfinished jobs (72 ms) 16:42:03 ✓ saves job when enqueued (32 ms) 16:42:03 ✓ does not retain job if database fails to save (4 ms) 16:42:03 ✓ saves job result (44 ms) 16:42:03 ✓ saves result even if database fails to save (37 ms) 16:42:03 ✓ saves job error (41 ms) 16:42:03 ✓ saves job error even if database fails to save (32 ms) 16:42:03 ✓ does not save job result if job is unknown (4 ms) 16:42:03 ✓ does not save job error if job is unknown (3 ms) 16:42:03 ✓ cleans up old jobs periodically (442 ms) 16:42:03 16:42:03 Test Suites: 1 passed, 1 total 16:42:03 Tests: 96 passed, 96 total 16:42:03 Snapshots: 0 total 16:42:03 Time: 13.647 s 16:42:03 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 16:42:03 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?