ACI ·

Parent Log: http://ci.aztec-labs.com/54f15b1cfed48c5d Command: 5fd30fafdf4fa09d yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/5c77169d7f01a9e15bc19b05fa231fd96111d7dd Env: REF_NAME=gh-readonly-queue/next/pr-15042-d5c48d855d83e19712c6b72281801c6fce176ec1 CURRENT_VERSION=0.87.6 CI_FULL=0 Date: Wed Jun 18 15:03:15 UTC 2025 System: ARCH=arm64 CPUS=64 MEM=247Gi HOSTNAME=pr-15042_arm64_a1-fast Resources: CPU_LIST=0-63 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 15:03:17 [15:03:17.892] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.897] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f8f6cc8e3f1d179c epochNumber=42 {"provingJobId":"1:BASE_PARITY:f8f6cc8e3f1d179c"} 15:03:17 [15:03:17.899] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:21da69fdb3cc3036 epochNumber=1 {"provingJobId":"1:BASE_PARITY:21da69fdb3cc3036"} 15:03:17 [15:03:17.924] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.925] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:11a3799ce16601a9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:11a3799ce16601a9"} 15:03:17 [15:03:17.926] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:45c26d5598b4bc6d epochNumber=1 {"provingJobId":"1:BASE_PARITY:45c26d5598b4bc6d"} 15:03:17 [15:03:17.928] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.929] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:864b9497ca9a8bac epochNumber=1 {"provingJobId":"1:BASE_PARITY:864b9497ca9a8bac"} 15:03:17 [15:03:17.930] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:864b9497ca9a8bac epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:864b9497ca9a8bac"} 15:03:17 [15:03:17.932] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.933] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:261ead739169fb31 epochNumber=1 {"provingJobId":"1:BASE_PARITY:261ead739169fb31"} 15:03:17 [15:03:17.934] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:261ead739169fb31 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:261ead739169fb31"} 15:03:17 [15:03:17.934] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:261ead739169fb31 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:261ead739169fb31"} 15:03:17 [15:03:17.935] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:261ead739169fb31 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:261ead739169fb31"} 15:03:17 [15:03:17.936] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.937] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:75bd1211b53b58c1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:75bd1211b53b58c1"} 15:03:17 [15:03:17.955] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.956] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.957] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:32e35c7abca1ff45 epochNumber=1 {"provingJobId":"1:BASE_PARITY:32e35c7abca1ff45"} 15:03:17 [15:03:17.957] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:32e35c7abca1ff45 {"provingJobId":"1:BASE_PARITY:32e35c7abca1ff45"} 15:03:17 [15:03:17.958] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:32e35c7abca1ff45 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:32e35c7abca1ff45"} 15:03:17 [15:03:17.958] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:32e35c7abca1ff45 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:32e35c7abca1ff45"} 15:03:17 [15:03:17.959] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.960] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6e3dd2bf240e23bd epochNumber=1 {"provingJobId":"1:BASE_PARITY:6e3dd2bf240e23bd"} 15:03:17 [15:03:17.960] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6e3dd2bf240e23bd {"provingJobId":"1:BASE_PARITY:6e3dd2bf240e23bd"} 15:03:17 [15:03:17.960] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6e3dd2bf240e23bd type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:6e3dd2bf240e23bd"} 15:03:17 [15:03:17.962] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.963] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4db73997f3b8df60 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4db73997f3b8df60"} 15:03:17 [15:03:17.963] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4db73997f3b8df60 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4db73997f3b8df60"} 15:03:17 [15:03:17.963] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4db73997f3b8df60 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:4db73997f3b8df60"} 15:03:17 [15:03:17.965] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.965] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3cb25f9fef148dde epochNumber=1 {"provingJobId":"1:BASE_PARITY:3cb25f9fef148dde"} 15:03:17 [15:03:17.965] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3cb25f9fef148dde type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3cb25f9fef148dde"} 15:03:17 [15:03:17.966] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3cb25f9fef148dde type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3cb25f9fef148dde"} 15:03:17 [15:03:17.967] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.967] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:85d42fa412760851 epochNumber=0 {"provingJobId":"1:BASE_PARITY:85d42fa412760851"} 15:03:17 [15:03:17.967] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:85d42fa412760851 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:85d42fa412760851"} 15:03:17 [15:03:17.968] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5b0e047340a711e2 epochNumber=0 {"provingJobId":"1:BASE_PARITY:5b0e047340a711e2"} 15:03:17 [15:03:17.968] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:85d42fa412760851 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:85d42fa412760851"} 15:03:17 [15:03:17.968] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e5d6c2be060cdf44 epochNumber=0 {"provingJobId":"1:BASE_PARITY:e5d6c2be060cdf44"} 15:03:17 [15:03:17.969] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.970] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.970] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b61c4043ba608b1a epochNumber=2 {"provingJobId":"1:BASE_PARITY:b61c4043ba608b1a"} 15:03:17 [15:03:17.971] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c832c7be3c992c8b epochNumber=1 {"provingJobId":"1:BASE_PARITY:c832c7be3c992c8b"} 15:03:17 [15:03:17.972] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.976] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6e65b46d9dbd18a2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6e65b46d9dbd18a2"} 15:03:17 [15:03:17.977] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.978] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c7f404a1112cbdff epochNumber=1 {"provingJobId":"1:BASE_PARITY:c7f404a1112cbdff"} 15:03:17 [15:03:17.978] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:10801a34e4f868fe epochNumber=1 {"provingJobId":"1:BASE_PARITY:10801a34e4f868fe"} 15:03:17 [15:03:17.978] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2924cbb5ae71e38b epochNumber=2 {"provingJobId":"1:BASE_PARITY:2924cbb5ae71e38b"} 15:03:17 [15:03:17.979] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:76208944200b7db2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:76208944200b7db2"} 15:03:17 [15:03:17.980] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.980] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e29497cc19ed2463 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e29497cc19ed2463"} 15:03:17 [15:03:17.980] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:409e200e23674b5d epochNumber=1 {"provingJobId":"1:BASE_PARITY:409e200e23674b5d"} 15:03:17 [15:03:17.981] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:17a05187dac2f998 epochNumber=2 {"provingJobId":"1:BASE_PARITY:17a05187dac2f998"} 15:03:17 [15:03:17.981] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:37f73425b83b4ec9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:37f73425b83b4ec9"} 15:03:17 [15:03:17.982] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.982] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fefe8faebdbde71b epochNumber=1 {"provingJobId":"1:BASE_PARITY:fefe8faebdbde71b"} 15:03:17 [15:03:17.982] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3d1a26c763920abb epochNumber=1 {"provingJobId":"1:BASE_PARITY:3d1a26c763920abb"} 15:03:17 [15:03:17.983] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f8ebc4c8d4c95d22 epochNumber=2 {"provingJobId":"1:BASE_PARITY:f8ebc4c8d4c95d22"} 15:03:17 [15:03:17.983] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:31b91920526c9d2d epochNumber=1 {"provingJobId":"1:BASE_PARITY:31b91920526c9d2d"} 15:03:17 [15:03:17.984] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.984] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ee0c2ca229c7d749 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ee0c2ca229c7d749"} 15:03:17 [15:03:17.985] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d964d813f9902074 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d964d813f9902074"} 15:03:17 [15:03:17.985] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ee0c2ca229c7d749 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ee0c2ca229c7d749"} 15:03:17 [15:03:17.986] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.986] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2e33f60559afd698 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2e33f60559afd698"} 15:03:17 [15:03:17.987] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:16869719aa535352 epochNumber=1 {"provingJobId":"1:BASE_PARITY:16869719aa535352"} 15:03:17 [15:03:17.987] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2e33f60559afd698 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:2e33f60559afd698"} 15:03:17 [15:03:17.988] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.988] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:29b9f5c7af215c0b epochNumber=1 {"provingJobId":"1:BASE_PARITY:29b9f5c7af215c0b"} 15:03:17 [15:03:17.989] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a19107c643c36cab epochNumber=1 {"provingJobId":"1:BASE_PARITY:a19107c643c36cab"} 15:03:17 [15:03:17.989] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:29b9f5c7af215c0b type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:29b9f5c7af215c0b"} 15:03:17 [15:03:17.990] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.990] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ac7af4de2b9ccb18 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ac7af4de2b9ccb18"} 15:03:17 [15:03:17.991] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:ac7af4de2b9ccb18 {"provingJobId":"1:BASE_PARITY:ac7af4de2b9ccb18"} 15:03:17 [15:03:17.991] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ac7af4de2b9ccb18 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:ac7af4de2b9ccb18"} 15:03:17 [15:03:17.991] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:531c513e4137b11c epochNumber=1 {"provingJobId":"1:BASE_PARITY:531c513e4137b11c"} 15:03:17 [15:03:17.992] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ac7af4de2b9ccb18 has already been completed {"provingJobId":"1:BASE_PARITY:ac7af4de2b9ccb18"} 15:03:17 [15:03:17.993] INFO: prover-client:proving-broker Proving Broker started 15:03:17 [15:03:17.993] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:41da55440d049eba epochNumber=1 {"provingJobId":"1:BASE_PARITY:41da55440d049eba"} 15:03:17 [15:03:17.993] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:76074cd51e3efebb epochNumber=2 {"provingJobId":"1:BASE_PARITY:76074cd51e3efebb"} 15:03:19 [15:03:19.045] INFO: prover-client:proving-broker Proving Broker started 15:03:19 [15:03:19.045] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:41da55440d049eba settled=false {"provingJobId":"1:BASE_PARITY:41da55440d049eba","status":"pending"} 15:03:19 [15:03:19.046] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:76074cd51e3efebb settled=false {"provingJobId":"1:BASE_PARITY:76074cd51e3efebb","status":"pending"} 15:03:19 [15:03:19.047] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:41da55440d049eba type=BASE_PARITY startedAt=1750258997994 older agent has taken job {"provingJobId":"1:BASE_PARITY:41da55440d049eba"} 15:03:19 [15:03:19.047] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:41da55440d049eba type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:41da55440d049eba"} 15:03:19 [15:03:19.048] INFO: prover-client:proving-broker Proving Broker started 15:03:19 [15:03:19.048] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6ee15a969b7c110f epochNumber=1 {"provingJobId":"1:BASE_PARITY:6ee15a969b7c110f"} 15:03:19 [15:03:19.049] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e8236fc61fc1450c epochNumber=2 {"provingJobId":"1:BASE_PARITY:e8236fc61fc1450c"} 15:03:20 [15:03:20.049] INFO: prover-client:proving-broker Proving Broker started 15:03:20 [15:03:20.050] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6ee15a969b7c110f settled=false {"provingJobId":"1:BASE_PARITY:6ee15a969b7c110f","status":"pending"} 15:03:20 [15:03:20.050] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e8236fc61fc1450c settled=false {"provingJobId":"1:BASE_PARITY:e8236fc61fc1450c","status":"pending"} 15:03:20 [15:03:20.051] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6ee15a969b7c110f type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:6ee15a969b7c110f"} 15:03:20 [15:03:20.052] INFO: prover-client:proving-broker Proving Broker started 15:03:20 [15:03:20.052] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2d85bd7fe675d30e epochNumber=1 {"provingJobId":"1:BASE_PARITY:2d85bd7fe675d30e"} 15:03:20 [15:03:20.052] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:848d7a6943f28f9e epochNumber=2 {"provingJobId":"1:BASE_PARITY:848d7a6943f28f9e"} 15:03:21 [15:03:21.053] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.054] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2d85bd7fe675d30e settled=false {"provingJobId":"1:BASE_PARITY:2d85bd7fe675d30e","status":"pending"} 15:03:21 [15:03:21.054] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:848d7a6943f28f9e settled=false {"provingJobId":"1:BASE_PARITY:848d7a6943f28f9e","status":"pending"} 15:03:21 [15:03:21.055] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2d85bd7fe675d30e type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2d85bd7fe675d30e"} 15:03:21 [15:03:21.055] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2d85bd7fe675d30e type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2d85bd7fe675d30e"} 15:03:21 [15:03:21.056] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.057] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a4a45ef911bb21fc epochNumber=1 {"provingJobId":"1:BASE_PARITY:a4a45ef911bb21fc"} 15:03:21 [15:03:21.057] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab4808cf9c65f151 epochNumber=2 {"provingJobId":"1:BASE_PARITY:ab4808cf9c65f151"} 15:03:21 [15:03:21.057] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a4a45ef911bb21fc type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:a4a45ef911bb21fc"} 15:03:21 [15:03:21.058] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ab4808cf9c65f151 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ab4808cf9c65f151"} 15:03:21 [15:03:21.059] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.059] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aa932d07570922cb epochNumber=1 {"provingJobId":"1:BASE_PARITY:aa932d07570922cb"} 15:03:21 [15:03:21.060] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b794de615b1e3995 epochNumber=2 {"provingJobId":"1:BASE_PARITY:b794de615b1e3995"} 15:03:21 [15:03:21.060] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:aa932d07570922cb type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:aa932d07570922cb"} 15:03:21 [15:03:21.060] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:aa932d07570922cb type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:aa932d07570922cb"} 15:03:21 [15:03:21.060] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b794de615b1e3995 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b794de615b1e3995"} 15:03:21 [15:03:21.061] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.062] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0529bfef2f5bd924 err=test error {"provingJoId":"1:BASE_PARITY:0529bfef2f5bd924"} 15:03:21 [15:03:21.063] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.063] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0079a7789d5f5780 not found {"provingJobId":"1:BASE_PARITY:0079a7789d5f5780"} 15:03:21 [15:03:21.064] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.065] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6b9fe2a12e8ed56b epochNumber=1 {"provingJobId":"1:BASE_PARITY:6b9fe2a12e8ed56b"} 15:03:21 [15:03:21.066] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.066] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:411175a603dc979d epochNumber=1 {"provingJobId":"1:BASE_PARITY:411175a603dc979d"} 15:03:21 [15:03:21.167] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:411175a603dc979d timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:411175a603dc979d"} 15:03:21 [15:03:21.174] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.174] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cd7603517b80a40e epochNumber=1 {"provingJobId":"1:BASE_PARITY:cd7603517b80a40e"} 15:03:21 [15:03:21.299] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cd7603517b80a40e timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:cd7603517b80a40e"} 15:03:21 [15:03:21.300] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8988d5e43b3745a5 epochNumber=10 {"provingJobId":"1:BASE_PARITY:8988d5e43b3745a5"} 15:03:21 [15:03:21.325] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:21 [15:03:21.427] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.428] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8a869caba4717b80 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8a869caba4717b80"} 15:03:21 [15:03:21.580] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8a869caba4717b80 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:8a869caba4717b80"} 15:03:21 [15:03:21.629] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.630] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:705934582132a2fe epochNumber=1 {"provingJobId":"1:BASE_PARITY:705934582132a2fe"} 15:03:21 [15:03:21.630] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:705934582132a2fe type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:705934582132a2fe"} 15:03:21 [15:03:21.631] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.632] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c06935953fd57eb9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c06935953fd57eb9"} 15:03:21 [15:03:21.633] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c06935953fd57eb9 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:c06935953fd57eb9"} 15:03:21 [15:03:21.633] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c06935953fd57eb9 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:c06935953fd57eb9"} 15:03:21 [15:03:21.634] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.634] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e333f8b9e4992f70 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e333f8b9e4992f70"} 15:03:21 [15:03:21.635] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e333f8b9e4992f70 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:e333f8b9e4992f70"} 15:03:21 [15:03:21.636] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.636] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f4b0c38697847405 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f4b0c38697847405"} 15:03:21 [15:03:21.637] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f4b0c38697847405 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:f4b0c38697847405"} 15:03:21 [15:03:21.637] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:139b92cdd740c626 epochNumber=3 {"provingJobId":"1:BASE_PARITY:139b92cdd740c626"} 15:03:21 [15:03:21.661] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:21 [15:03:21.662] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:f4b0c38697847405 err=test error {"provingJoId":"1:BASE_PARITY:f4b0c38697847405"} 15:03:21 [15:03:21.664] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.664] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:32162cfee03c9209 settled=false {"provingJobId":"1:BASE_PARITY:32162cfee03c9209","status":"pending"} 15:03:21 [15:03:21.664] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6793b2a3c5fe8fb8 settled=false {"provingJobId":"1:BASE_PARITY:6793b2a3c5fe8fb8","status":"pending"} 15:03:21 [15:03:21.666] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.667] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:df387c92da6b8469 settled=true {"provingJobId":"1:BASE_PARITY:df387c92da6b8469","status":"fulfilled"} 15:03:21 [15:03:21.667] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:3dfd60c2b6da59b7 settled=true {"provingJobId":"2:BASE_PARITY:3dfd60c2b6da59b7","status":"fulfilled"} 15:03:21 [15:03:21.668] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.668] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6ea94ec293885b01 settled=true {"provingJobId":"1:BASE_PARITY:6ea94ec293885b01","status":"fulfilled"} 15:03:21 [15:03:21.668] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9413f97e63b2f2b5 settled=false {"provingJobId":"1:BASE_PARITY:9413f97e63b2f2b5","status":"pending"} 15:03:21 [15:03:21.670] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.671] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:343dbba243bee5e4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:343dbba243bee5e4"} 15:03:21 [15:03:21.672] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.673] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c6ae64522fe3e567 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c6ae64522fe3e567"} 15:03:21 [15:03:21.673] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:c6ae64522fe3e567: Error: db error: Error: db error 15:03:21 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:c6ae64522fe3e567"} 15:03:21 [15:03:21.674] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.675] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:94c8e918ce1f71d4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:94c8e918ce1f71d4"} 15:03:21 [15:03:21.675] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:94c8e918ce1f71d4 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:94c8e918ce1f71d4"} 15:03:21 [15:03:21.675] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:94c8e918ce1f71d4 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:94c8e918ce1f71d4"} 15:03:21 [15:03:21.676] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.677] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:521724df1bed1509 epochNumber=1 {"provingJobId":"1:BASE_PARITY:521724df1bed1509"} 15:03:21 [15:03:21.677] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:521724df1bed1509 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:521724df1bed1509"} 15:03:21 [15:03:21.677] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:521724df1bed1509 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:521724df1bed1509"} 15:03:21 [15:03:21.678] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:521724df1bed1509: Error: db error 15:03:21 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:521724df1bed1509"} 15:03:21 [15:03:21.679] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.679] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9e220612d3700163 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9e220612d3700163"} 15:03:21 [15:03:21.680] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9e220612d3700163 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:9e220612d3700163"} 15:03:21 [15:03:21.680] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9e220612d3700163 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9e220612d3700163"} 15:03:21 [15:03:21.681] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.681] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3c10f221424aed24 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3c10f221424aed24"} 15:03:21 [15:03:21.682] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3c10f221424aed24 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3c10f221424aed24"} 15:03:21 [15:03:21.682] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3c10f221424aed24 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3c10f221424aed24"} 15:03:21 [15:03:21.682] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:3c10f221424aed24 jobErr=test error: Error: db error 15:03:21 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:3c10f221424aed24"} 15:03:21 [15:03:21.686] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.686] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ceb6957493f19d84 not found {"provingJobId":"1:BASE_PARITY:ceb6957493f19d84"} 15:03:21 [15:03:21.687] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.688] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:8248ebf8f24a2213 err=test error {"provingJoId":"1:BASE_PARITY:8248ebf8f24a2213"} 15:03:21 [15:03:21.689] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.700] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:27cd7d09dd4c1fb4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:27cd7d09dd4c1fb4"} 15:03:21 [15:03:21.701] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:27cd7d09dd4c1fb4 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:27cd7d09dd4c1fb4"} 15:03:21 [15:03:21.701] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:27cd7d09dd4c1fb4 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:27cd7d09dd4c1fb4"} 15:03:21 [15:03:21.711] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:304885c85f574ae5 epochNumber=2 {"provingJobId":"2:BASE_PARITY:304885c85f574ae5"} 15:03:21 [15:03:21.711] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:304885c85f574ae5 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:304885c85f574ae5"} 15:03:21 [15:03:21.711] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:304885c85f574ae5 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:304885c85f574ae5"} 15:03:21 [15:03:21.772] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:411c2555d1f06076 epochNumber=3 {"provingJobId":"3:BASE_PARITY:411c2555d1f06076"} 15:03:21 [15:03:21.790] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:21 [15:03:21.833] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:e80a6eb1908ace3b epochNumber=4 {"provingJobId":"4:BASE_PARITY:e80a6eb1908ace3b"} 15:03:21 [15:03:21.840] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:21 [15:03:21.894] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:014c320383fbdb4e epochNumber=5 {"provingJobId":"5:BASE_PARITY:014c320383fbdb4e"} 15:03:21 [15:03:21.917] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:21 [15:03:21.947] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.948] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b0bf77997991810b epochNumber=42 {"provingJobId":"1:BASE_PARITY:b0bf77997991810b"} 15:03:21 [15:03:21.949] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testr4uSJM/42 with map size 1073741824 15:03:21 [15:03:21.951] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:21 [15:03:21.979] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:4d18d2e3ae24185b epochNumber=1 {"provingJobId":"1:BASE_PARITY:4d18d2e3ae24185b"} 15:03:21 [15:03:21.983] INFO: prover-client:proving-broker Proving Broker started 15:03:21 [15:03:21.983] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:05446027a09849b0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:05446027a09849b0"} 15:03:21 [15:03:21.985] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKj7ypZ/1 with map size 1073741824 15:03:21 [15:03:21.986] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.003] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2dda41df6da87f30 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2dda41df6da87f30"} 15:03:22 [15:03:22.010] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.011] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:02979f06fc0c517c epochNumber=1 {"provingJobId":"1:BASE_PARITY:02979f06fc0c517c"} 15:03:22 [15:03:22.011] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-teste8aNrC/1 with map size 1073741824 15:03:22 [15:03:22.012] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.029] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:02979f06fc0c517c epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:02979f06fc0c517c"} 15:03:22 [15:03:22.032] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.032] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:be7db5d4e7bebde9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:be7db5d4e7bebde9"} 15:03:22 [15:03:22.033] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4Wo955/1 with map size 1073741824 15:03:22 [15:03:22.034] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.050] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:be7db5d4e7bebde9 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:be7db5d4e7bebde9"} 15:03:22 [15:03:22.050] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:be7db5d4e7bebde9 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:be7db5d4e7bebde9"} 15:03:22 [15:03:22.057] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:be7db5d4e7bebde9 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:be7db5d4e7bebde9"} 15:03:22 [15:03:22.061] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.062] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:622afb8326117ae2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:622afb8326117ae2"} 15:03:22 [15:03:22.062] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUzk823/1 with map size 1073741824 15:03:22 [15:03:22.063] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.084] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.087] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.088] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:035a420bc734122a epochNumber=1 {"provingJobId":"1:BASE_PARITY:035a420bc734122a"} 15:03:22 [15:03:22.088] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDOUdMc/1 with map size 1073741824 15:03:22 [15:03:22.091] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.107] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:035a420bc734122a {"provingJobId":"1:BASE_PARITY:035a420bc734122a"} 15:03:22 [15:03:22.108] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:035a420bc734122a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:035a420bc734122a"} 15:03:22 [15:03:22.108] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:035a420bc734122a type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:035a420bc734122a"} 15:03:22 [15:03:22.115] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.116] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0507ed68e1cdac3e epochNumber=1 {"provingJobId":"1:BASE_PARITY:0507ed68e1cdac3e"} 15:03:22 [15:03:22.116] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testl2Lhxg/1 with map size 1073741824 15:03:22 [15:03:22.117] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.132] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0507ed68e1cdac3e {"provingJobId":"1:BASE_PARITY:0507ed68e1cdac3e"} 15:03:22 [15:03:22.133] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0507ed68e1cdac3e type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:0507ed68e1cdac3e"} 15:03:22 [15:03:22.142] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.143] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2b640907c71eba46 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2b640907c71eba46"} 15:03:22 [15:03:22.145] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmva7fD/1 with map size 1073741824 15:03:22 [15:03:22.147] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.162] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2b640907c71eba46 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2b640907c71eba46"} 15:03:22 [15:03:22.162] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2b640907c71eba46 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2b640907c71eba46"} 15:03:22 [15:03:22.169] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.170] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03350978a8e857ce epochNumber=1 {"provingJobId":"1:BASE_PARITY:03350978a8e857ce"} 15:03:22 [15:03:22.170] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testafjSSv/1 with map size 1073741824 15:03:22 [15:03:22.172] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.187] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:03350978a8e857ce type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:03350978a8e857ce"} 15:03:22 [15:03:22.187] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:03350978a8e857ce type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:03350978a8e857ce"} 15:03:22 [15:03:22.193] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.194] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1362d9b7e57d6ea6 epochNumber=0 {"provingJobId":"1:BASE_PARITY:1362d9b7e57d6ea6"} 15:03:22 [15:03:22.195] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testtbMdI9/0 with map size 1073741824 15:03:22 [15:03:22.196] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.210] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1362d9b7e57d6ea6 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:1362d9b7e57d6ea6"} 15:03:22 [15:03:22.211] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:556631ca59d8f697 epochNumber=0 {"provingJobId":"1:BASE_PARITY:556631ca59d8f697"} 15:03:22 [15:03:22.211] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1362d9b7e57d6ea6 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:1362d9b7e57d6ea6"} 15:03:22 [15:03:22.211] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:69728528f947b0b5 epochNumber=0 {"provingJobId":"1:BASE_PARITY:69728528f947b0b5"} 15:03:22 [15:03:22.226] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.227] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.228] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:82940b660c78be5a epochNumber=2 {"provingJobId":"1:BASE_PARITY:82940b660c78be5a"} 15:03:22 [15:03:22.228] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testS7wBgR/2 with map size 1073741824 15:03:22 [15:03:22.229] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.253] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:187764f4ed4390e4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:187764f4ed4390e4"} 15:03:22 [15:03:22.253] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testS7wBgR/1 with map size 1073741824 15:03:22 [15:03:22.255] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.277] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.278] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a67420aa6d30e97a epochNumber=1 {"provingJobId":"1:BASE_PARITY:a67420aa6d30e97a"} 15:03:22 [15:03:22.278] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOwhbb4/1 with map size 1073741824 15:03:22 [15:03:22.280] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.302] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.303] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6bea4400291337ca epochNumber=1 {"provingJobId":"1:BASE_PARITY:6bea4400291337ca"} 15:03:22 [15:03:22.303] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUJ7N8z/1 with map size 1073741824 15:03:22 [15:03:22.305] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.322] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:304f5d5c3e3c2a1c epochNumber=1 {"provingJobId":"1:BASE_PARITY:304f5d5c3e3c2a1c"} 15:03:22 [15:03:22.327] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d11da145dc09cf22 epochNumber=2 {"provingJobId":"1:BASE_PARITY:d11da145dc09cf22"} 15:03:22 [15:03:22.328] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUJ7N8z/2 with map size 1073741824 15:03:22 [15:03:22.329] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.345] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:19463810e5007f15 epochNumber=1 {"provingJobId":"1:BASE_PARITY:19463810e5007f15"} 15:03:22 [15:03:22.352] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.353] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a33e0059ab01e0c5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a33e0059ab01e0c5"} 15:03:22 [15:03:22.353] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyAgjHh/1 with map size 1073741824 15:03:22 [15:03:22.356] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.370] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c7c6d2a0405a6c55 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c7c6d2a0405a6c55"} 15:03:22 [15:03:22.375] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b9a0e8ed8ef6a2fe epochNumber=2 {"provingJobId":"1:BASE_PARITY:b9a0e8ed8ef6a2fe"} 15:03:22 [15:03:22.376] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testyAgjHh/2 with map size 1073741824 15:03:22 [15:03:22.377] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.392] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:389f753ee3b0fb97 epochNumber=1 {"provingJobId":"1:BASE_PARITY:389f753ee3b0fb97"} 15:03:22 [15:03:22.400] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.400] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bf33611e9d808e0e epochNumber=1 {"provingJobId":"1:BASE_PARITY:bf33611e9d808e0e"} 15:03:22 [15:03:22.401] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeJg0a3/1 with map size 1073741824 15:03:22 [15:03:22.402] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.416] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:757135ac8661724b epochNumber=1 {"provingJobId":"1:BASE_PARITY:757135ac8661724b"} 15:03:22 [15:03:22.422] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1279d509a82ee8c4 epochNumber=2 {"provingJobId":"1:BASE_PARITY:1279d509a82ee8c4"} 15:03:22 [15:03:22.422] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testeJg0a3/2 with map size 1073741824 15:03:22 [15:03:22.424] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.438] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6dc701d79a96b44d epochNumber=1 {"provingJobId":"1:BASE_PARITY:6dc701d79a96b44d"} 15:03:22 [15:03:22.445] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.445] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b108fd284f0d9cca epochNumber=1 {"provingJobId":"1:BASE_PARITY:b108fd284f0d9cca"} 15:03:22 [15:03:22.446] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfLyPES/1 with map size 1073741824 15:03:22 [15:03:22.447] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.461] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:37e78cab8b5b8201 epochNumber=1 {"provingJobId":"1:BASE_PARITY:37e78cab8b5b8201"} 15:03:22 [15:03:22.466] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b108fd284f0d9cca type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b108fd284f0d9cca"} 15:03:22 [15:03:22.473] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.474] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4d53c46d7bb71cfd epochNumber=1 {"provingJobId":"1:BASE_PARITY:4d53c46d7bb71cfd"} 15:03:22 [15:03:22.474] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmKt1Zo/1 with map size 1073741824 15:03:22 [15:03:22.475] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.490] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:638c1f53ba24a6c5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:638c1f53ba24a6c5"} 15:03:22 [15:03:22.495] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4d53c46d7bb71cfd type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:4d53c46d7bb71cfd"} 15:03:22 [15:03:22.502] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.503] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dfc47f4514c7ebea epochNumber=1 {"provingJobId":"1:BASE_PARITY:dfc47f4514c7ebea"} 15:03:22 [15:03:22.504] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbGvolS/1 with map size 1073741824 15:03:22 [15:03:22.506] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.524] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:963d750956d14496 epochNumber=1 {"provingJobId":"1:BASE_PARITY:963d750956d14496"} 15:03:22 [15:03:22.529] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:dfc47f4514c7ebea type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:dfc47f4514c7ebea"} 15:03:22 [15:03:22.533] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.533] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:00854a0ead754fdd epochNumber=1 {"provingJobId":"1:BASE_PARITY:00854a0ead754fdd"} 15:03:22 [15:03:22.534] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDPNm0k/1 with map size 1073741824 15:03:22 [15:03:22.535] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.553] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:00854a0ead754fdd {"provingJobId":"1:BASE_PARITY:00854a0ead754fdd"} 15:03:22 [15:03:22.554] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:00854a0ead754fdd type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:00854a0ead754fdd"} 15:03:22 [15:03:22.561] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e7d0acd95b7438d9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e7d0acd95b7438d9"} 15:03:22 [15:03:22.568] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:00854a0ead754fdd has already been completed {"provingJobId":"1:BASE_PARITY:00854a0ead754fdd"} 15:03:22 [15:03:22.571] INFO: prover-client:proving-broker Proving Broker started 15:03:22 [15:03:22.572] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:88f779baa4e37452 epochNumber=1 {"provingJobId":"1:BASE_PARITY:88f779baa4e37452"} 15:03:22 [15:03:22.572] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCo5ZnW/1 with map size 1073741824 15:03:22 [15:03:22.573] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:22 [15:03:22.589] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e96a8b9ab31a6918 epochNumber=2 {"provingJobId":"1:BASE_PARITY:e96a8b9ab31a6918"} 15:03:22 [15:03:22.590] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCo5ZnW/2 with map size 1073741824 15:03:22 [15:03:22.591] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:23 [15:03:23.656] INFO: prover-client:proving-broker Proving Broker started 15:03:23 [15:03:23.660] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:88f779baa4e37452 settled=false {"provingJobId":"1:BASE_PARITY:88f779baa4e37452","status":"pending"} 15:03:23 [15:03:23.662] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e96a8b9ab31a6918 settled=false {"provingJobId":"1:BASE_PARITY:e96a8b9ab31a6918","status":"pending"} 15:03:23 [15:03:23.663] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:88f779baa4e37452 type=BASE_PARITY startedAt=1750259002606 older agent has taken job {"provingJobId":"1:BASE_PARITY:88f779baa4e37452"} 15:03:23 [15:03:23.663] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:88f779baa4e37452 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:88f779baa4e37452"} 15:03:23 [15:03:23.666] INFO: prover-client:proving-broker Proving Broker started 15:03:23 [15:03:23.667] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:76f1b52e929bc995 epochNumber=1 {"provingJobId":"1:BASE_PARITY:76f1b52e929bc995"} 15:03:23 [15:03:23.667] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQcuD1Y/1 with map size 1073741824 15:03:23 [15:03:23.668] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:23 [15:03:23.684] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ad0ed3cab01dc124 epochNumber=2 {"provingJobId":"1:BASE_PARITY:ad0ed3cab01dc124"} 15:03:23 [15:03:23.684] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQcuD1Y/2 with map size 1073741824 15:03:23 [15:03:23.686] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:24 [15:03:24.702] INFO: prover-client:proving-broker Proving Broker started 15:03:24 [15:03:24.704] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:76f1b52e929bc995 settled=false {"provingJobId":"1:BASE_PARITY:76f1b52e929bc995","status":"pending"} 15:03:24 [15:03:24.705] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ad0ed3cab01dc124 settled=false {"provingJobId":"1:BASE_PARITY:ad0ed3cab01dc124","status":"pending"} 15:03:24 [15:03:24.706] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:76f1b52e929bc995 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:76f1b52e929bc995"} 15:03:24 [15:03:24.709] INFO: prover-client:proving-broker Proving Broker started 15:03:24 [15:03:24.710] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:25880cf67600bbce epochNumber=1 {"provingJobId":"1:BASE_PARITY:25880cf67600bbce"} 15:03:24 [15:03:24.710] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVe5a9f/1 with map size 1073741824 15:03:24 [15:03:24.711] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:24 [15:03:24.727] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a0641d8e57230037 epochNumber=2 {"provingJobId":"1:BASE_PARITY:a0641d8e57230037"} 15:03:24 [15:03:24.728] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVe5a9f/2 with map size 1073741824 15:03:24 [15:03:24.730] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:25 [15:03:25.745] INFO: prover-client:proving-broker Proving Broker started 15:03:25 [15:03:25.747] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:25880cf67600bbce settled=false {"provingJobId":"1:BASE_PARITY:25880cf67600bbce","status":"pending"} 15:03:25 [15:03:25.748] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a0641d8e57230037 settled=false {"provingJobId":"1:BASE_PARITY:a0641d8e57230037","status":"pending"} 15:03:25 [15:03:25.749] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:25880cf67600bbce type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:25880cf67600bbce"} 15:03:25 [15:03:25.749] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:25880cf67600bbce type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:25880cf67600bbce"} 15:03:25 [15:03:25.757] INFO: prover-client:proving-broker Proving Broker started 15:03:25 [15:03:25.758] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1fd54eb62679954b epochNumber=1 {"provingJobId":"1:BASE_PARITY:1fd54eb62679954b"} 15:03:25 [15:03:25.758] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAsqls6/1 with map size 1073741824 15:03:25 [15:03:25.759] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:25 [15:03:25.774] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8b25d4491a21d6c5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:8b25d4491a21d6c5"} 15:03:25 [15:03:25.774] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testAsqls6/2 with map size 1073741824 15:03:25 [15:03:25.776] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:25 [15:03:25.789] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1fd54eb62679954b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:1fd54eb62679954b"} 15:03:25 [15:03:25.795] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8b25d4491a21d6c5 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:8b25d4491a21d6c5"} 15:03:25 [15:03:25.802] INFO: prover-client:proving-broker Proving Broker started 15:03:25 [15:03:25.803] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e0b2e0e32c22dd8b epochNumber=1 {"provingJobId":"1:BASE_PARITY:e0b2e0e32c22dd8b"} 15:03:25 [15:03:25.803] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkxbt73/1 with map size 1073741824 15:03:25 [15:03:25.804] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:25 [15:03:25.819] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3c3f7e9424817a54 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3c3f7e9424817a54"} 15:03:25 [15:03:25.819] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkxbt73/2 with map size 1073741824 15:03:25 [15:03:25.821] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:25 [15:03:25.834] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e0b2e0e32c22dd8b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e0b2e0e32c22dd8b"} 15:03:25 [15:03:25.835] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e0b2e0e32c22dd8b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e0b2e0e32c22dd8b"} 15:03:25 [15:03:25.840] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3c3f7e9424817a54 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3c3f7e9424817a54"} 15:03:25 [15:03:25.847] INFO: prover-client:proving-broker Proving Broker started 15:03:25 [15:03:25.848] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6073b16c6e826703 err=test error {"provingJoId":"1:BASE_PARITY:6073b16c6e826703"} 15:03:25 [15:03:25.851] INFO: prover-client:proving-broker Proving Broker started 15:03:25 [15:03:25.852] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:aeb2f650aedfc831 not found {"provingJobId":"1:BASE_PARITY:aeb2f650aedfc831"} 15:03:25 [15:03:25.854] INFO: prover-client:proving-broker Proving Broker started 15:03:25 [15:03:25.855] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:06325ac7bf0c6580 epochNumber=1 {"provingJobId":"1:BASE_PARITY:06325ac7bf0c6580"} 15:03:25 [15:03:25.855] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMzAxlN/1 with map size 1073741824 15:03:25 [15:03:25.857] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:25 [15:03:25.874] INFO: prover-client:proving-broker Proving Broker started 15:03:25 [15:03:25.874] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1371a38cae200e31 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1371a38cae200e31"} 15:03:25 [15:03:25.875] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testohYanY/1 with map size 1073741824 15:03:25 [15:03:25.876] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:25 [15:03:25.998] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1371a38cae200e31 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:1371a38cae200e31"} 15:03:26 [15:03:26.006] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.007] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:621fbf11bc606d8f epochNumber=1 {"provingJobId":"1:BASE_PARITY:621fbf11bc606d8f"} 15:03:26 [15:03:26.007] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testicfmAU/1 with map size 1073741824 15:03:26 [15:03:26.009] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.132] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:621fbf11bc606d8f timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:621fbf11bc606d8f"} 15:03:26 [15:03:26.137] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8be37cb1196663ab epochNumber=10 {"provingJobId":"1:BASE_PARITY:8be37cb1196663ab"} 15:03:26 [15:03:26.138] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testicfmAU/10 with map size 1073741824 15:03:26 [15:03:26.139] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.157] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:26 [15:03:26.157] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 15:03:26 [15:03:26.282] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.282] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ba0e647543419df epochNumber=1 {"provingJobId":"1:BASE_PARITY:8ba0e647543419df"} 15:03:26 [15:03:26.283] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnkn9FB/1 with map size 1073741824 15:03:26 [15:03:26.284] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.459] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8ba0e647543419df timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:8ba0e647543419df"} 15:03:26 [15:03:26.502] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.502] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:22ab4fb6f7ca374e epochNumber=1 {"provingJobId":"1:BASE_PARITY:22ab4fb6f7ca374e"} 15:03:26 [15:03:26.503] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxjPCNr/1 with map size 1073741824 15:03:26 [15:03:26.504] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.518] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:22ab4fb6f7ca374e type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:22ab4fb6f7ca374e"} 15:03:26 [15:03:26.521] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.522] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d2d45fcb0a9349db epochNumber=1 {"provingJobId":"1:BASE_PARITY:d2d45fcb0a9349db"} 15:03:26 [15:03:26.522] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgt0fB7/1 with map size 1073741824 15:03:26 [15:03:26.524] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.537] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d2d45fcb0a9349db type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:d2d45fcb0a9349db"} 15:03:26 [15:03:26.538] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d2d45fcb0a9349db type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:d2d45fcb0a9349db"} 15:03:26 [15:03:26.545] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cf57502bb93733c8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cf57502bb93733c8"} 15:03:26 [15:03:26.546] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHfzOUG/1 with map size 1073741824 15:03:26 [15:03:26.547] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.561] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cf57502bb93733c8 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:cf57502bb93733c8"} 15:03:26 [15:03:26.568] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.569] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3bd61d19b4b9b9cb epochNumber=1 {"provingJobId":"1:BASE_PARITY:3bd61d19b4b9b9cb"} 15:03:26 [15:03:26.569] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxq5r8Q/1 with map size 1073741824 15:03:26 [15:03:26.570] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.584] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3bd61d19b4b9b9cb type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:3bd61d19b4b9b9cb"} 15:03:26 [15:03:26.585] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7a934568c9a95c90 epochNumber=3 {"provingJobId":"1:BASE_PARITY:7a934568c9a95c90"} 15:03:26 [15:03:26.585] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testxq5r8Q/3 with map size 1073741824 15:03:26 [15:03:26.587] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.618] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:26 [15:03:26.618] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 15:03:26 [15:03:26.626] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3bd61d19b4b9b9cb err=test error {"provingJoId":"1:BASE_PARITY:3bd61d19b4b9b9cb"} 15:03:26 [15:03:26.629] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGve9jr/1 with map size 1073741824 15:03:26 [15:03:26.631] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.645] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGve9jr/2 with map size 1073741824 15:03:26 [15:03:26.646] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.660] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.661] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:92d2c3d8f472f562 settled=false {"provingJobId":"1:BASE_PARITY:92d2c3d8f472f562","status":"pending"} 15:03:26 [15:03:26.663] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:12a30438b767b10e settled=false {"provingJobId":"1:BASE_PARITY:12a30438b767b10e","status":"pending"} 15:03:26 [15:03:26.668] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testU1396c/1 with map size 1073741824 15:03:26 [15:03:26.669] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.684] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testU1396c/2 with map size 1073741824 15:03:26 [15:03:26.685] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.711] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.712] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8c227d147bf1aa8c settled=true {"provingJobId":"1:BASE_PARITY:8c227d147bf1aa8c","status":"fulfilled"} 15:03:26 [15:03:26.714] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:daf2e9f63f75dcc6 settled=true {"provingJobId":"2:BASE_PARITY:daf2e9f63f75dcc6","status":"fulfilled"} 15:03:26 [15:03:26.718] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testykJ78f/1 with map size 1073741824 15:03:26 [15:03:26.720] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.739] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testykJ78f/2 with map size 1073741824 15:03:26 [15:03:26.741] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.756] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.758] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4c140d65aa79e4ca settled=true {"provingJobId":"1:BASE_PARITY:4c140d65aa79e4ca","status":"fulfilled"} 15:03:26 [15:03:26.760] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:87d6d1ca52546791 settled=false {"provingJobId":"1:BASE_PARITY:87d6d1ca52546791","status":"pending"} 15:03:26 [15:03:26.763] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.764] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0d019224327e7c08 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0d019224327e7c08"} 15:03:26 [15:03:26.764] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testRFpGkP/1 with map size 1073741824 15:03:26 [15:03:26.766] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.783] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.783] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:23ac28552eb1d58d epochNumber=1 {"provingJobId":"1:BASE_PARITY:23ac28552eb1d58d"} 15:03:26 [15:03:26.784] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:23ac28552eb1d58d: Error: db error: Error: db error 15:03:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:23ac28552eb1d58d"} 15:03:26 [15:03:26.788] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.788] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3b4230f6fc184d7c epochNumber=1 {"provingJobId":"1:BASE_PARITY:3b4230f6fc184d7c"} 15:03:26 [15:03:26.789] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMKJOWQ/1 with map size 1073741824 15:03:26 [15:03:26.790] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.805] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3b4230f6fc184d7c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3b4230f6fc184d7c"} 15:03:26 [15:03:26.806] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3b4230f6fc184d7c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3b4230f6fc184d7c"} 15:03:26 [15:03:26.814] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.815] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f2abb5b06a60fc36 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f2abb5b06a60fc36"} 15:03:26 [15:03:26.816] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrroqmT/1 with map size 1073741824 15:03:26 [15:03:26.817] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.832] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f2abb5b06a60fc36 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f2abb5b06a60fc36"} 15:03:26 [15:03:26.833] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f2abb5b06a60fc36 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f2abb5b06a60fc36"} 15:03:26 [15:03:26.833] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:f2abb5b06a60fc36: Error: db error 15:03:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:f2abb5b06a60fc36"} 15:03:26 [15:03:26.837] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.837] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bebf8aa5989ba79e epochNumber=1 {"provingJobId":"1:BASE_PARITY:bebf8aa5989ba79e"} 15:03:26 [15:03:26.838] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKnnj6j/1 with map size 1073741824 15:03:26 [15:03:26.839] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.859] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bebf8aa5989ba79e type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:bebf8aa5989ba79e"} 15:03:26 [15:03:26.860] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bebf8aa5989ba79e type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:bebf8aa5989ba79e"} 15:03:26 [15:03:26.872] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.873] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:152b50d7300f2692 epochNumber=1 {"provingJobId":"1:BASE_PARITY:152b50d7300f2692"} 15:03:26 [15:03:26.873] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testejb7sA/1 with map size 1073741824 15:03:26 [15:03:26.875] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.892] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:152b50d7300f2692 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:152b50d7300f2692"} 15:03:26 [15:03:26.893] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:152b50d7300f2692 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:152b50d7300f2692"} 15:03:26 [15:03:26.893] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:152b50d7300f2692 jobErr=test error: Error: db error 15:03:26 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:152b50d7300f2692"} 15:03:26 [15:03:26.897] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.897] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4af047aa90981a40 not found {"provingJobId":"1:BASE_PARITY:4af047aa90981a40"} 15:03:26 [15:03:26.900] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.901] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ffb108440b3bc6d8 err=test error {"provingJoId":"1:BASE_PARITY:ffb108440b3bc6d8"} 15:03:26 [15:03:26.904] INFO: prover-client:proving-broker Proving Broker started 15:03:26 [15:03:26.914] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d71624126f83b717 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d71624126f83b717"} 15:03:26 [15:03:26.915] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcp0aps/1 with map size 1073741824 15:03:26 [15:03:26.916] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.935] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d71624126f83b717 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d71624126f83b717"} 15:03:26 [15:03:26.936] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d71624126f83b717 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d71624126f83b717"} 15:03:26 [15:03:26.952] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:6237f7d4bd73ee84 epochNumber=2 {"provingJobId":"2:BASE_PARITY:6237f7d4bd73ee84"} 15:03:26 [15:03:26.953] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testcp0aps/2 with map size 1073741824 15:03:26 [15:03:26.954] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:26 [15:03:26.972] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:6237f7d4bd73ee84 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:6237f7d4bd73ee84"} 15:03:26 [15:03:26.973] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:6237f7d4bd73ee84 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:6237f7d4bd73ee84"} 15:03:27 [15:03:27.039] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:505dfda0486e3f76 epochNumber=3 {"provingJobId":"3:BASE_PARITY:505dfda0486e3f76"} 15:03:27 [15:03:27.040] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testcp0aps/3 with map size 1073741824 15:03:27 [15:03:27.042] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:27 [15:03:27.079] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:27 [15:03:27.080] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 15:03:27 [15:03:27.117] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:60b267e2a93f5282 epochNumber=4 {"provingJobId":"4:BASE_PARITY:60b267e2a93f5282"} 15:03:27 [15:03:27.118] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testcp0aps/4 with map size 1073741824 15:03:27 [15:03:27.119] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:27 [15:03:27.157] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:27 [15:03:27.158] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 15:03:27 [15:03:27.197] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:96e761bc3bac8e68 epochNumber=5 {"provingJobId":"5:BASE_PARITY:96e761bc3bac8e68"} 15:03:27 [15:03:27.198] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testcp0aps/5 with map size 1073741824 15:03:27 [15:03:27.200] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 15:03:27 [15:03:27.234] INFO: prover-client:proving-broker Cleaned up jobs=1 15:03:27 [15:03:27.235] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 15:03:27 PASS src/proving_broker/proving_broker.test.ts (11.245 s) 15:03:27 ProvingBroker 15:03:27 Producer API 15:03:27 ✓ refuses stale jobs (35 ms) 15:03:27 ✓ enqueues jobs (4 ms) 15:03:27 ✓ ignores duplicate jobs (3 ms) 15:03:27 ✓ reports correct status when enqueuing repeat jobs (4 ms) 15:03:27 ✓ throws an error in case of duplicate job IDs (18 ms) 15:03:27 ✓ returns not-found status for non-existing jobs (2 ms) 15:03:27 ✓ cancels jobs in queue (3 ms) 15:03:27 ✓ cancels jobs in-progress (2 ms) 15:03:27 ✓ returns job result if successful (3 ms) 15:03:27 ✓ returns job error if failed (2 ms) 15:03:27 ✓ correctly returns job status for concurrent writes (2 ms) 15:03:27 ✓ refuses stale jobs (36 ms) 15:03:27 ✓ enqueues jobs (27 ms) 15:03:27 ✓ ignores duplicate jobs (21 ms) 15:03:27 ✓ reports correct status when enqueuing repeat jobs (27 ms) 15:03:27 ✓ throws an error in case of duplicate job IDs (23 ms) 15:03:27 ✓ returns not-found status for non-existing jobs (3 ms) 15:03:27 ✓ cancels jobs in queue (29 ms) 15:03:27 ✓ cancels jobs in-progress (26 ms) 15:03:27 ✓ returns job result if successful (28 ms) 15:03:27 ✓ returns job error if failed (24 ms) 15:03:27 ✓ correctly returns job status for concurrent writes (31 ms) 15:03:27 Consumer API 15:03:27 ✓ returns undefined if no jobs are available (1 ms) 15:03:27 ✓ returns jobs in priority order (1 ms) 15:03:27 ✓ returns undefined if no jobs are available for the given allowList (6 ms) 15:03:27 ✓ returns a job if it is in the allowList (2 ms) 15:03:27 ✓ returns the most important job if it is in the allowList (2 ms) 15:03:27 ✓ returns any job if filter is empty (2 ms) 15:03:27 ✓ returns a new job when reporting job success (2 ms) 15:03:27 ✓ returns a new job when reporting permanent error (2 ms) 15:03:27 ✓ returns a new job when reporting retry-able error (2 ms) 15:03:27 ✓ returns a new job when reporting progress if current one is cancelled (2 ms) 15:03:27 ✓ returns a new job if job is already in progress elsewhere (1055 ms) 15:03:27 ✓ avoids sending the same job to a new agent after a restart (1003 ms) 15:03:27 ✓ avoids sending a completed job to a new agent after a restart (1005 ms) 15:03:27 ✓ tracks job result if in progress (2 ms) 15:03:27 ✓ tracks job result even if job is in queue (2 ms) 15:03:27 ✓ ignores reported job error if unknown job (1 ms) 15:03:27 ✓ ignores job result if unknown job (2 ms) 15:03:27 ✓ returns undefined if no jobs are available (1 ms) 15:03:27 ✓ returns jobs in priority order (50 ms) 15:03:27 ✓ returns undefined if no jobs are available for the given allowList (25 ms) 15:03:27 ✓ returns a job if it is in the allowList (50 ms) 15:03:27 ✓ returns the most important job if it is in the allowList (47 ms) 15:03:27 ✓ returns any job if filter is empty (46 ms) 15:03:27 ✓ returns a new job when reporting job success (28 ms) 15:03:27 ✓ returns a new job when reporting permanent error (29 ms) 15:03:27 ✓ returns a new job when reporting retry-able error (30 ms) 15:03:27 ✓ returns a new job when reporting progress if current one is cancelled (38 ms) 15:03:27 ✓ returns a new job if job is already in progress elsewhere (1094 ms) 15:03:27 ✓ avoids sending the same job to a new agent after a restart (1042 ms) 15:03:27 ✓ avoids sending a completed job to a new agent after a restart (1048 ms) 15:03:27 ✓ tracks job result if in progress (44 ms) 15:03:27 ✓ tracks job result even if job is in queue (45 ms) 15:03:27 ✓ ignores reported job error if unknown job (3 ms) 15:03:27 ✓ ignores job result if unknown job (4 ms) 15:03:27 Timeouts 15:03:27 ✓ tracks in progress jobs (1 ms) 15:03:27 ✓ re-enqueues jobs that time out (107 ms) 15:03:27 ✓ cancel stale jobs that time out (253 ms) 15:03:27 ✓ keeps the jobs in progress while it is alive (201 ms) 15:03:27 ✓ tracks in progress jobs (19 ms) 15:03:27 ✓ re-enqueues jobs that time out (132 ms) 15:03:27 ✓ cancel stale jobs that time out (275 ms) 15:03:27 ✓ keeps the jobs in progress while it is alive (219 ms) 15:03:27 Retries 15:03:27 ✓ retries jobs (2 ms) 15:03:27 ✓ retries up to a maximum number of times (2 ms) 15:03:27 ✓ passing retry=false does not retry (1 ms) 15:03:27 ✓ does not retry if job is stale (28 ms) 15:03:27 ✓ retries jobs (19 ms) 15:03:27 ✓ retries up to a maximum number of times (23 ms) 15:03:27 ✓ passing retry=false does not retry (22 ms) 15:03:27 ✓ does not retry if job is stale (60 ms) 15:03:27 Database management 15:03:27 ✓ re-enqueues proof requests on start (3 ms) 15:03:27 ✓ restores proof results on start (1 ms) 15:03:27 ✓ only re-enqueues unfinished jobs (1 ms) 15:03:27 ✓ saves job when enqueued (3 ms) 15:03:27 ✓ does not retain job if database fails to save (2 ms) 15:03:27 ✓ saves job result (2 ms) 15:03:27 ✓ saves result even if database fails to save (2 ms) 15:03:27 ✓ saves job error (3 ms) 15:03:27 ✓ saves job error even if database fails to save (2 ms) 15:03:27 ✓ does not save job result if job is unknown (4 ms) 15:03:27 ✓ does not save job error if job is unknown (2 ms) 15:03:27 ✓ cleans up old jobs periodically (256 ms) 15:03:27 ✓ re-enqueues proof requests on start (39 ms) 15:03:27 ✓ restores proof results on start (50 ms) 15:03:27 ✓ only re-enqueues unfinished jobs (46 ms) 15:03:27 ✓ saves job when enqueued (19 ms) 15:03:27 ✓ does not retain job if database fails to save (4 ms) 15:03:27 ✓ saves job result (27 ms) 15:03:27 ✓ saves result even if database fails to save (22 ms) 15:03:27 ✓ saves job error (33 ms) 15:03:27 ✓ saves job error even if database fails to save (26 ms) 15:03:27 ✓ does not save job result if job is unknown (3 ms) 15:03:27 ✓ does not save job error if job is unknown (3 ms) 15:03:27 ✓ cleans up old jobs periodically (365 ms) 15:03:27 15:03:27 Test Suites: 1 passed, 1 total 15:03:27 Tests: 96 passed, 96 total 15:03:27 Snapshots: 0 total 15:03:27 Time: 11.318 s 15:03:27 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 15:03:27 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?