Parent Log: http://ci.aztec-labs.com/cda42aa9421eefa5 Command: 7c288cdaeed3e186 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/58b067708e8ff927bf26b2994203134853a7eb71 Env: REF_NAME=gh-readonly-queue/next/pr-15078-3b44f825dcd3249b2c8a1105905fa3394aa7213d CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Mon Jun 16 17:53:50 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15078_amd64_x1-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 17:53:54 [17:53:54.508] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.514] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ed4686067cd01b2e epochNumber=42 {"provingJobId":"1:BASE_PARITY:ed4686067cd01b2e"} 17:53:54 [17:53:54.518] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:b17a1ff3114d641e epochNumber=1 {"provingJobId":"1:BASE_PARITY:b17a1ff3114d641e"} 17:53:54 [17:53:54.566] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.567] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2b660ae438304d8d epochNumber=1 {"provingJobId":"1:BASE_PARITY:2b660ae438304d8d"} 17:53:54 [17:53:54.568] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:41b0396ed23c4fc2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:41b0396ed23c4fc2"} 17:53:54 [17:53:54.570] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.571] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6c50464825bb933c epochNumber=1 {"provingJobId":"1:BASE_PARITY:6c50464825bb933c"} 17:53:54 [17:53:54.572] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6c50464825bb933c epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:6c50464825bb933c"} 17:53:54 [17:53:54.574] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.575] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6b2164f337eecc63 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6b2164f337eecc63"} 17:53:54 [17:53:54.576] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6b2164f337eecc63 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:6b2164f337eecc63"} 17:53:54 [17:53:54.576] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6b2164f337eecc63 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6b2164f337eecc63"} 17:53:54 [17:53:54.577] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6b2164f337eecc63 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:6b2164f337eecc63"} 17:53:54 [17:53:54.578] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.578] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d484b96faa951845 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d484b96faa951845"} 17:53:54 [17:53:54.601] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.605] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.606] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fc6a38ff1aff616e epochNumber=1 {"provingJobId":"1:BASE_PARITY:fc6a38ff1aff616e"} 17:53:54 [17:53:54.608] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:fc6a38ff1aff616e {"provingJobId":"1:BASE_PARITY:fc6a38ff1aff616e"} 17:53:54 [17:53:54.608] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:fc6a38ff1aff616e type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:fc6a38ff1aff616e"} 17:53:54 [17:53:54.609] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fc6a38ff1aff616e type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:fc6a38ff1aff616e"} 17:53:54 [17:53:54.611] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.612] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:84330dece1622ac5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:84330dece1622ac5"} 17:53:54 [17:53:54.613] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:84330dece1622ac5 {"provingJobId":"1:BASE_PARITY:84330dece1622ac5"} 17:53:54 [17:53:54.613] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:84330dece1622ac5 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:84330dece1622ac5"} 17:53:54 [17:53:54.616] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.618] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:66f2bb38f8acbbdb epochNumber=1 {"provingJobId":"1:BASE_PARITY:66f2bb38f8acbbdb"} 17:53:54 [17:53:54.619] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:66f2bb38f8acbbdb type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:66f2bb38f8acbbdb"} 17:53:54 [17:53:54.620] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:66f2bb38f8acbbdb type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:66f2bb38f8acbbdb"} 17:53:54 [17:53:54.623] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.624] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7e3fcd026c49e3e4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7e3fcd026c49e3e4"} 17:53:54 [17:53:54.624] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7e3fcd026c49e3e4 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:7e3fcd026c49e3e4"} 17:53:54 [17:53:54.625] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7e3fcd026c49e3e4 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:7e3fcd026c49e3e4"} 17:53:54 [17:53:54.626] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.626] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae6b7853c16255e6 epochNumber=0 {"provingJobId":"1:BASE_PARITY:ae6b7853c16255e6"} 17:53:54 [17:53:54.627] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ae6b7853c16255e6 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:ae6b7853c16255e6"} 17:53:54 [17:53:54.627] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:04ff4ae271d0f016 epochNumber=0 {"provingJobId":"1:BASE_PARITY:04ff4ae271d0f016"} 17:53:54 [17:53:54.627] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:ae6b7853c16255e6 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:ae6b7853c16255e6"} 17:53:54 [17:53:54.628] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a08465e330bc0ad9 epochNumber=0 {"provingJobId":"1:BASE_PARITY:a08465e330bc0ad9"} 17:53:54 [17:53:54.629] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.630] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.630] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e171b288b3391a2e epochNumber=2 {"provingJobId":"1:BASE_PARITY:e171b288b3391a2e"} 17:53:54 [17:53:54.630] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:59a9c6b80c8188cd epochNumber=1 {"provingJobId":"1:BASE_PARITY:59a9c6b80c8188cd"} 17:53:54 [17:53:54.632] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.632] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:896e5005cf32823b epochNumber=1 {"provingJobId":"1:BASE_PARITY:896e5005cf32823b"} 17:53:54 [17:53:54.633] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.633] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f1e2cf59a9cc9ab2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f1e2cf59a9cc9ab2"} 17:53:54 [17:53:54.633] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dbef02685a67018f epochNumber=1 {"provingJobId":"1:BASE_PARITY:dbef02685a67018f"} 17:53:54 [17:53:54.634] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bbc4175e651ccdd1 epochNumber=2 {"provingJobId":"1:BASE_PARITY:bbc4175e651ccdd1"} 17:53:54 [17:53:54.634] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:70ee19dcd0b971b5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:70ee19dcd0b971b5"} 17:53:54 [17:53:54.635] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.635] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:962084ff895b0132 epochNumber=1 {"provingJobId":"1:BASE_PARITY:962084ff895b0132"} 17:53:54 [17:53:54.635] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea21a7d016954df8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea21a7d016954df8"} 17:53:54 [17:53:54.635] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a42980f8b0876aae epochNumber=2 {"provingJobId":"1:BASE_PARITY:a42980f8b0876aae"} 17:53:54 [17:53:54.636] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c6f37ff1c48a7adf epochNumber=1 {"provingJobId":"1:BASE_PARITY:c6f37ff1c48a7adf"} 17:53:54 [17:53:54.636] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.637] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9fbd57142055fc9f epochNumber=1 {"provingJobId":"1:BASE_PARITY:9fbd57142055fc9f"} 17:53:54 [17:53:54.637] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fcb75be4b4955381 epochNumber=1 {"provingJobId":"1:BASE_PARITY:fcb75be4b4955381"} 17:53:54 [17:53:54.637] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:82f0a4475e3c01de epochNumber=2 {"provingJobId":"1:BASE_PARITY:82f0a4475e3c01de"} 17:53:54 [17:53:54.637] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2558929f491715b2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2558929f491715b2"} 17:53:54 [17:53:54.638] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.639] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a40f77c9ffa8e339 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a40f77c9ffa8e339"} 17:53:54 [17:53:54.639] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4982e25d4dbb78d8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4982e25d4dbb78d8"} 17:53:54 [17:53:54.639] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:a40f77c9ffa8e339 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:a40f77c9ffa8e339"} 17:53:54 [17:53:54.640] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.641] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0ec4b5d242c19a75 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0ec4b5d242c19a75"} 17:53:54 [17:53:54.641] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:38ec0b1c92c9798b epochNumber=1 {"provingJobId":"1:BASE_PARITY:38ec0b1c92c9798b"} 17:53:54 [17:53:54.641] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0ec4b5d242c19a75 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:0ec4b5d242c19a75"} 17:53:54 [17:53:54.642] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.642] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c2530acc3572a8db epochNumber=1 {"provingJobId":"1:BASE_PARITY:c2530acc3572a8db"} 17:53:54 [17:53:54.643] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cebb3cb4e1086e31 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cebb3cb4e1086e31"} 17:53:54 [17:53:54.643] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c2530acc3572a8db type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:c2530acc3572a8db"} 17:53:54 [17:53:54.644] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.644] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9712b5aa2b8459f5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9712b5aa2b8459f5"} 17:53:54 [17:53:54.644] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:9712b5aa2b8459f5 {"provingJobId":"1:BASE_PARITY:9712b5aa2b8459f5"} 17:53:54 [17:53:54.644] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9712b5aa2b8459f5 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:9712b5aa2b8459f5"} 17:53:54 [17:53:54.645] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ee458fce0c0872ea epochNumber=1 {"provingJobId":"1:BASE_PARITY:ee458fce0c0872ea"} 17:53:54 [17:53:54.645] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9712b5aa2b8459f5 has already been completed {"provingJobId":"1:BASE_PARITY:9712b5aa2b8459f5"} 17:53:54 [17:53:54.654] INFO: prover-client:proving-broker Proving Broker started 17:53:54 [17:53:54.655] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f2c0f5e74bf574eb epochNumber=1 {"provingJobId":"1:BASE_PARITY:f2c0f5e74bf574eb"} 17:53:54 [17:53:54.656] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:99f6a781e09bcf1c epochNumber=2 {"provingJobId":"1:BASE_PARITY:99f6a781e09bcf1c"} 17:53:55 [17:53:55.709] INFO: prover-client:proving-broker Proving Broker started 17:53:55 [17:53:55.710] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f2c0f5e74bf574eb settled=false {"provingJobId":"1:BASE_PARITY:f2c0f5e74bf574eb","status":"pending"} 17:53:55 [17:53:55.710] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:99f6a781e09bcf1c settled=false {"provingJobId":"1:BASE_PARITY:99f6a781e09bcf1c","status":"pending"} 17:53:55 [17:53:55.711] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:f2c0f5e74bf574eb type=BASE_PARITY startedAt=1750096434656 older agent has taken job {"provingJobId":"1:BASE_PARITY:f2c0f5e74bf574eb"} 17:53:55 [17:53:55.711] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f2c0f5e74bf574eb type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:f2c0f5e74bf574eb"} 17:53:55 [17:53:55.713] INFO: prover-client:proving-broker Proving Broker started 17:53:55 [17:53:55.714] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e8a5090bd82e9751 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e8a5090bd82e9751"} 17:53:55 [17:53:55.714] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:57a7bacf341cfc7f epochNumber=2 {"provingJobId":"1:BASE_PARITY:57a7bacf341cfc7f"} 17:53:56 [17:53:56.714] INFO: prover-client:proving-broker Proving Broker started 17:53:56 [17:53:56.715] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e8a5090bd82e9751 settled=false {"provingJobId":"1:BASE_PARITY:e8a5090bd82e9751","status":"pending"} 17:53:56 [17:53:56.715] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:57a7bacf341cfc7f settled=false {"provingJobId":"1:BASE_PARITY:57a7bacf341cfc7f","status":"pending"} 17:53:56 [17:53:56.716] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e8a5090bd82e9751 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:e8a5090bd82e9751"} 17:53:56 [17:53:56.718] INFO: prover-client:proving-broker Proving Broker started 17:53:56 [17:53:56.718] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:136e56f47c610670 epochNumber=1 {"provingJobId":"1:BASE_PARITY:136e56f47c610670"} 17:53:56 [17:53:56.719] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ef1f367f5e83facb epochNumber=2 {"provingJobId":"1:BASE_PARITY:ef1f367f5e83facb"} 17:53:57 [17:53:57.720] INFO: prover-client:proving-broker Proving Broker started 17:53:57 [17:53:57.721] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:136e56f47c610670 settled=false {"provingJobId":"1:BASE_PARITY:136e56f47c610670","status":"pending"} 17:53:57 [17:53:57.721] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ef1f367f5e83facb settled=false {"provingJobId":"1:BASE_PARITY:ef1f367f5e83facb","status":"pending"} 17:53:57 [17:53:57.724] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:136e56f47c610670 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:136e56f47c610670"} 17:53:57 [17:53:57.724] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:136e56f47c610670 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:136e56f47c610670"} 17:53:57 [17:53:57.728] INFO: prover-client:proving-broker Proving Broker started 17:53:57 [17:53:57.729] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9b7958f1d9bc8b41 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9b7958f1d9bc8b41"} 17:53:57 [17:53:57.730] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f79ad75b59882826 epochNumber=2 {"provingJobId":"1:BASE_PARITY:f79ad75b59882826"} 17:53:57 [17:53:57.730] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9b7958f1d9bc8b41 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:9b7958f1d9bc8b41"} 17:53:57 [17:53:57.731] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f79ad75b59882826 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:f79ad75b59882826"} 17:53:57 [17:53:57.732] INFO: prover-client:proving-broker Proving Broker started 17:53:57 [17:53:57.733] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2a6d1aad3ed1d591 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2a6d1aad3ed1d591"} 17:53:57 [17:53:57.733] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:59f09671918c681d epochNumber=2 {"provingJobId":"1:BASE_PARITY:59f09671918c681d"} 17:53:57 [17:53:57.734] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2a6d1aad3ed1d591 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2a6d1aad3ed1d591"} 17:53:57 [17:53:57.734] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2a6d1aad3ed1d591 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2a6d1aad3ed1d591"} 17:53:57 [17:53:57.734] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:59f09671918c681d type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:59f09671918c681d"} 17:53:57 [17:53:57.735] INFO: prover-client:proving-broker Proving Broker started 17:53:57 [17:53:57.736] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:b341bb0879189803 err=test error {"provingJoId":"1:BASE_PARITY:b341bb0879189803"} 17:53:57 [17:53:57.737] INFO: prover-client:proving-broker Proving Broker started 17:53:57 [17:53:57.738] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:551b3188d53d0d89 not found {"provingJobId":"1:BASE_PARITY:551b3188d53d0d89"} 17:53:57 [17:53:57.744] INFO: prover-client:proving-broker Proving Broker started 17:53:57 [17:53:57.746] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d2a1ff1ee0695696 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d2a1ff1ee0695696"} 17:53:57 [17:53:57.748] INFO: prover-client:proving-broker Proving Broker started 17:53:57 [17:53:57.748] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5be1e54ab25fc9b7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5be1e54ab25fc9b7"} 17:53:57 [17:53:57.850] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5be1e54ab25fc9b7 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:5be1e54ab25fc9b7"} 17:53:57 [17:53:57.857] INFO: prover-client:proving-broker Proving Broker started 17:53:57 [17:53:57.858] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:be5fceed268e40bf epochNumber=1 {"provingJobId":"1:BASE_PARITY:be5fceed268e40bf"} 17:53:57 [17:53:57.960] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:be5fceed268e40bf timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:be5fceed268e40bf"} 17:53:57 [17:53:57.966] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8f22269a1f68f3c8 epochNumber=10 {"provingJobId":"1:BASE_PARITY:8f22269a1f68f3c8"} 17:53:57 [17:53:57.985] INFO: prover-client:proving-broker Cleaned up jobs=1 17:53:58 [17:53:58.093] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.094] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5e063e16751e5ab6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5e063e16751e5ab6"} 17:53:58 [17:53:58.248] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5e063e16751e5ab6 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:5e063e16751e5ab6"} 17:53:58 [17:53:58.300] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.306] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:149add1e134aca75 epochNumber=1 {"provingJobId":"1:BASE_PARITY:149add1e134aca75"} 17:53:58 [17:53:58.307] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:149add1e134aca75 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:149add1e134aca75"} 17:53:58 [17:53:58.309] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.310] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f21b1a98270fd521 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f21b1a98270fd521"} 17:53:58 [17:53:58.311] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f21b1a98270fd521 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:f21b1a98270fd521"} 17:53:58 [17:53:58.311] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f21b1a98270fd521 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:f21b1a98270fd521"} 17:53:58 [17:53:58.313] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.313] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f677fb6368b3e4c8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f677fb6368b3e4c8"} 17:53:58 [17:53:58.318] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f677fb6368b3e4c8 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:f677fb6368b3e4c8"} 17:53:58 [17:53:58.320] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.321] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:954eb0acfd39def6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:954eb0acfd39def6"} 17:53:58 [17:53:58.322] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:954eb0acfd39def6 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:954eb0acfd39def6"} 17:53:58 [17:53:58.323] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e6ab6c342e5524f9 epochNumber=3 {"provingJobId":"1:BASE_PARITY:e6ab6c342e5524f9"} 17:53:58 [17:53:58.346] INFO: prover-client:proving-broker Cleaned up jobs=1 17:53:58 [17:53:58.348] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:954eb0acfd39def6 err=test error {"provingJoId":"1:BASE_PARITY:954eb0acfd39def6"} 17:53:58 [17:53:58.350] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.351] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4c212826dd84d9cf settled=false {"provingJobId":"1:BASE_PARITY:4c212826dd84d9cf","status":"pending"} 17:53:58 [17:53:58.352] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4f8e13a07a04cf1f settled=false {"provingJobId":"1:BASE_PARITY:4f8e13a07a04cf1f","status":"pending"} 17:53:58 [17:53:58.355] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.356] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5350ebfe18470013 settled=true {"provingJobId":"1:BASE_PARITY:5350ebfe18470013","status":"fulfilled"} 17:53:58 [17:53:58.357] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:7e2ac964ce5a0cd7 settled=true {"provingJobId":"2:BASE_PARITY:7e2ac964ce5a0cd7","status":"fulfilled"} 17:53:58 [17:53:58.359] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.360] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c0d205879d512383 settled=true {"provingJobId":"1:BASE_PARITY:c0d205879d512383","status":"fulfilled"} 17:53:58 [17:53:58.360] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:46c1eab817d938b1 settled=false {"provingJobId":"1:BASE_PARITY:46c1eab817d938b1","status":"pending"} 17:53:58 [17:53:58.362] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.365] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:20627f19046cdd6d epochNumber=1 {"provingJobId":"1:BASE_PARITY:20627f19046cdd6d"} 17:53:58 [17:53:58.368] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.369] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5b8f8456bf652549 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5b8f8456bf652549"} 17:53:58 [17:53:58.370] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:5b8f8456bf652549: Error: db error: Error: db error 17:53:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:5b8f8456bf652549"} 17:53:58 [17:53:58.372] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.373] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f30b2872251f7834 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f30b2872251f7834"} 17:53:58 [17:53:58.373] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f30b2872251f7834 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f30b2872251f7834"} 17:53:58 [17:53:58.373] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f30b2872251f7834 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f30b2872251f7834"} 17:53:58 [17:53:58.375] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.375] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5d2de35f8a6a19a9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5d2de35f8a6a19a9"} 17:53:58 [17:53:58.376] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5d2de35f8a6a19a9 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5d2de35f8a6a19a9"} 17:53:58 [17:53:58.376] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5d2de35f8a6a19a9 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5d2de35f8a6a19a9"} 17:53:58 [17:53:58.377] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:5d2de35f8a6a19a9: Error: db error 17:53:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:5d2de35f8a6a19a9"} 17:53:58 [17:53:58.379] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.379] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:adb71a2f5545e467 epochNumber=1 {"provingJobId":"1:BASE_PARITY:adb71a2f5545e467"} 17:53:58 [17:53:58.380] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:adb71a2f5545e467 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:adb71a2f5545e467"} 17:53:58 [17:53:58.380] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:adb71a2f5545e467 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:adb71a2f5545e467"} 17:53:58 [17:53:58.382] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.382] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:823ebe54c044b99b epochNumber=1 {"provingJobId":"1:BASE_PARITY:823ebe54c044b99b"} 17:53:58 [17:53:58.382] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:823ebe54c044b99b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:823ebe54c044b99b"} 17:53:58 [17:53:58.383] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:823ebe54c044b99b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:823ebe54c044b99b"} 17:53:58 [17:53:58.383] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:823ebe54c044b99b jobErr=test error: Error: db error 17:53:58 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:823ebe54c044b99b"} 17:53:58 [17:53:58.385] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.385] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a99b312471a8a700 not found {"provingJobId":"1:BASE_PARITY:a99b312471a8a700"} 17:53:58 [17:53:58.387] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.388] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:ecb348b42e15a4ca err=test error {"provingJoId":"1:BASE_PARITY:ecb348b42e15a4ca"} 17:53:58 [17:53:58.389] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.401] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8d0595b509ff2275 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8d0595b509ff2275"} 17:53:58 [17:53:58.402] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8d0595b509ff2275 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8d0595b509ff2275"} 17:53:58 [17:53:58.402] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8d0595b509ff2275 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8d0595b509ff2275"} 17:53:58 [17:53:58.413] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:5dfd29667e853bb0 epochNumber=2 {"provingJobId":"2:BASE_PARITY:5dfd29667e853bb0"} 17:53:58 [17:53:58.414] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:5dfd29667e853bb0 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:5dfd29667e853bb0"} 17:53:58 [17:53:58.414] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:5dfd29667e853bb0 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:5dfd29667e853bb0"} 17:53:58 [17:53:58.475] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:613a441390950181 epochNumber=3 {"provingJobId":"3:BASE_PARITY:613a441390950181"} 17:53:58 [17:53:58.493] INFO: prover-client:proving-broker Cleaned up jobs=1 17:53:58 [17:53:58.537] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:dc562bd35e2636ed epochNumber=4 {"provingJobId":"4:BASE_PARITY:dc562bd35e2636ed"} 17:53:58 [17:53:58.545] INFO: prover-client:proving-broker Cleaned up jobs=1 17:53:58 [17:53:58.600] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:8f409302f59a7f3a epochNumber=5 {"provingJobId":"5:BASE_PARITY:8f409302f59a7f3a"} 17:53:58 [17:53:58.622] INFO: prover-client:proving-broker Cleaned up jobs=1 17:53:58 [17:53:58.655] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.656] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:91bcf53ee1c7ebdc epochNumber=42 {"provingJobId":"1:BASE_PARITY:91bcf53ee1c7ebdc"} 17:53:58 [17:53:58.657] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testEUkPWl/42 with map size 1073741824 17:53:58 [17:53:58.662] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:58 [17:53:58.691] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:5646b57b9c75e3f6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5646b57b9c75e3f6"} 17:53:58 [17:53:58.699] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.700] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:129c759a1514a405 epochNumber=1 {"provingJobId":"1:BASE_PARITY:129c759a1514a405"} 17:53:58 [17:53:58.701] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwbHGA8/1 with map size 1073741824 17:53:58 [17:53:58.703] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:58 [17:53:58.724] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fb93583b162265b1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:fb93583b162265b1"} 17:53:58 [17:53:58.735] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.735] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:62296546625b74b5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:62296546625b74b5"} 17:53:58 [17:53:58.736] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLB375J/1 with map size 1073741824 17:53:58 [17:53:58.739] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:58 [17:53:58.762] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:62296546625b74b5 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:62296546625b74b5"} 17:53:58 [17:53:58.766] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.766] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:56cd6ebcc21e8a2f epochNumber=1 {"provingJobId":"1:BASE_PARITY:56cd6ebcc21e8a2f"} 17:53:58 [17:53:58.767] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQd79vn/1 with map size 1073741824 17:53:58 [17:53:58.768] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:58 [17:53:58.788] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:56cd6ebcc21e8a2f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:56cd6ebcc21e8a2f"} 17:53:58 [17:53:58.789] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:56cd6ebcc21e8a2f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:56cd6ebcc21e8a2f"} 17:53:58 [17:53:58.797] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:56cd6ebcc21e8a2f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:56cd6ebcc21e8a2f"} 17:53:58 [17:53:58.811] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.812] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6c333cef9a3cdb77 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6c333cef9a3cdb77"} 17:53:58 [17:53:58.814] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYNFrPA/1 with map size 1073741824 17:53:58 [17:53:58.818] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:58 [17:53:58.843] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.847] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.847] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2991f6c85b71a839 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2991f6c85b71a839"} 17:53:58 [17:53:58.848] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFNHKVD/1 with map size 1073741824 17:53:58 [17:53:58.850] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:58 [17:53:58.867] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2991f6c85b71a839 {"provingJobId":"1:BASE_PARITY:2991f6c85b71a839"} 17:53:58 [17:53:58.868] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2991f6c85b71a839 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2991f6c85b71a839"} 17:53:58 [17:53:58.868] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2991f6c85b71a839 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:2991f6c85b71a839"} 17:53:58 [17:53:58.879] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.879] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c10cc7c534d4439f epochNumber=1 {"provingJobId":"1:BASE_PARITY:c10cc7c534d4439f"} 17:53:58 [17:53:58.880] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjdwgvu/1 with map size 1073741824 17:53:58 [17:53:58.882] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:58 [17:53:58.905] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c10cc7c534d4439f {"provingJobId":"1:BASE_PARITY:c10cc7c534d4439f"} 17:53:58 [17:53:58.906] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c10cc7c534d4439f type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:c10cc7c534d4439f"} 17:53:58 [17:53:58.916] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.918] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:17ee3394e71139c1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:17ee3394e71139c1"} 17:53:58 [17:53:58.918] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWsXKLr/1 with map size 1073741824 17:53:58 [17:53:58.920] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:58 [17:53:58.940] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:17ee3394e71139c1 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:17ee3394e71139c1"} 17:53:58 [17:53:58.941] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:17ee3394e71139c1 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:17ee3394e71139c1"} 17:53:58 [17:53:58.949] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.950] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:20613b0cd30124ba epochNumber=1 {"provingJobId":"1:BASE_PARITY:20613b0cd30124ba"} 17:53:58 [17:53:58.951] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testM9cn9Y/1 with map size 1073741824 17:53:58 [17:53:58.953] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:58 [17:53:58.973] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:20613b0cd30124ba type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:20613b0cd30124ba"} 17:53:58 [17:53:58.974] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:20613b0cd30124ba type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:20613b0cd30124ba"} 17:53:58 [17:53:58.983] INFO: prover-client:proving-broker Proving Broker started 17:53:58 [17:53:58.984] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2a150b2fa17bd299 epochNumber=0 {"provingJobId":"1:BASE_PARITY:2a150b2fa17bd299"} 17:53:58 [17:53:58.984] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test0xBnOg/0 with map size 1073741824 17:53:58 [17:53:58.987] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.012] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2a150b2fa17bd299 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:2a150b2fa17bd299"} 17:53:59 [17:53:59.013] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2c0a5b854b352be1 epochNumber=0 {"provingJobId":"1:BASE_PARITY:2c0a5b854b352be1"} 17:53:59 [17:53:59.014] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2a150b2fa17bd299 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:2a150b2fa17bd299"} 17:53:59 [17:53:59.014] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b2e2d160df0e5952 epochNumber=0 {"provingJobId":"1:BASE_PARITY:b2e2d160df0e5952"} 17:53:59 [17:53:59.033] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.035] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.035] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f0c3c3c49a6f0aa5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:f0c3c3c49a6f0aa5"} 17:53:59 [17:53:59.036] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testn2C78y/2 with map size 1073741824 17:53:59 [17:53:59.038] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.057] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a2b60ac65d6ce315 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a2b60ac65d6ce315"} 17:53:59 [17:53:59.058] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testn2C78y/1 with map size 1073741824 17:53:59 [17:53:59.061] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.085] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.086] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f54e878f6ba4ba5a epochNumber=1 {"provingJobId":"1:BASE_PARITY:f54e878f6ba4ba5a"} 17:53:59 [17:53:59.086] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testelXDKH/1 with map size 1073741824 17:53:59 [17:53:59.091] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.117] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.117] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bf0440604688fd73 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bf0440604688fd73"} 17:53:59 [17:53:59.118] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGQd8hR/1 with map size 1073741824 17:53:59 [17:53:59.121] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.138] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:932013da3ca6af28 epochNumber=1 {"provingJobId":"1:BASE_PARITY:932013da3ca6af28"} 17:53:59 [17:53:59.145] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cd3e157843946d8a epochNumber=2 {"provingJobId":"1:BASE_PARITY:cd3e157843946d8a"} 17:53:59 [17:53:59.146] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testGQd8hR/2 with map size 1073741824 17:53:59 [17:53:59.147] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.187] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:24d1e9f9ebe2355a epochNumber=1 {"provingJobId":"1:BASE_PARITY:24d1e9f9ebe2355a"} 17:53:59 [17:53:59.193] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.194] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:352c42448bee5465 epochNumber=1 {"provingJobId":"1:BASE_PARITY:352c42448bee5465"} 17:53:59 [17:53:59.198] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testl8F2gr/1 with map size 1073741824 17:53:59 [17:53:59.201] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.221] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f1470f44120d79cb epochNumber=1 {"provingJobId":"1:BASE_PARITY:f1470f44120d79cb"} 17:53:59 [17:53:59.227] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d582c224007fcf67 epochNumber=2 {"provingJobId":"1:BASE_PARITY:d582c224007fcf67"} 17:53:59 [17:53:59.228] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testl8F2gr/2 with map size 1073741824 17:53:59 [17:53:59.230] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.249] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:685149b41cf81384 epochNumber=1 {"provingJobId":"1:BASE_PARITY:685149b41cf81384"} 17:53:59 [17:53:59.259] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.259] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:69be5aa4bbd8d6ae epochNumber=1 {"provingJobId":"1:BASE_PARITY:69be5aa4bbd8d6ae"} 17:53:59 [17:53:59.260] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtWKIaP/1 with map size 1073741824 17:53:59 [17:53:59.261] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.281] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3e31209d57f89367 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3e31209d57f89367"} 17:53:59 [17:53:59.288] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1f0e6ab2d7e5824c epochNumber=2 {"provingJobId":"1:BASE_PARITY:1f0e6ab2d7e5824c"} 17:53:59 [17:53:59.289] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testtWKIaP/2 with map size 1073741824 17:53:59 [17:53:59.292] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.312] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4d193fca2aa94882 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4d193fca2aa94882"} 17:53:59 [17:53:59.322] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.323] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9e1e77cb70f63759 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9e1e77cb70f63759"} 17:53:59 [17:53:59.323] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkzIk6x/1 with map size 1073741824 17:53:59 [17:53:59.325] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.345] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f16921fa9cae3ec0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f16921fa9cae3ec0"} 17:53:59 [17:53:59.353] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:9e1e77cb70f63759 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:9e1e77cb70f63759"} 17:53:59 [17:53:59.368] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.369] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0495c1cd18ce47db epochNumber=1 {"provingJobId":"1:BASE_PARITY:0495c1cd18ce47db"} 17:53:59 [17:53:59.371] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testn0zOz3/1 with map size 1073741824 17:53:59 [17:53:59.374] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.393] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c0bbfb2b89f2068e epochNumber=1 {"provingJobId":"1:BASE_PARITY:c0bbfb2b89f2068e"} 17:53:59 [17:53:59.399] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0495c1cd18ce47db type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:0495c1cd18ce47db"} 17:53:59 [17:53:59.408] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.409] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:36b0f6a8183a339b epochNumber=1 {"provingJobId":"1:BASE_PARITY:36b0f6a8183a339b"} 17:53:59 [17:53:59.410] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGQGXPP/1 with map size 1073741824 17:53:59 [17:53:59.414] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.436] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a10781d1cd61c8ad epochNumber=1 {"provingJobId":"1:BASE_PARITY:a10781d1cd61c8ad"} 17:53:59 [17:53:59.443] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:36b0f6a8183a339b type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:36b0f6a8183a339b"} 17:53:59 [17:53:59.448] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.449] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:643e856e8701965e epochNumber=1 {"provingJobId":"1:BASE_PARITY:643e856e8701965e"} 17:53:59 [17:53:59.450] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7ZbmTF/1 with map size 1073741824 17:53:59 [17:53:59.452] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.471] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:643e856e8701965e {"provingJobId":"1:BASE_PARITY:643e856e8701965e"} 17:53:59 [17:53:59.471] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:643e856e8701965e type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:643e856e8701965e"} 17:53:59 [17:53:59.477] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:baa02fd822acb3e2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:baa02fd822acb3e2"} 17:53:59 [17:53:59.483] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:643e856e8701965e has already been completed {"provingJobId":"1:BASE_PARITY:643e856e8701965e"} 17:53:59 [17:53:59.489] INFO: prover-client:proving-broker Proving Broker started 17:53:59 [17:53:59.490] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9022622c43ec8c86 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9022622c43ec8c86"} 17:53:59 [17:53:59.491] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFrzxDy/1 with map size 1073741824 17:53:59 [17:53:59.496] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:53:59 [17:53:59.523] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:95683d87dcd51a5f epochNumber=2 {"provingJobId":"1:BASE_PARITY:95683d87dcd51a5f"} 17:53:59 [17:53:59.524] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFrzxDy/2 with map size 1073741824 17:53:59 [17:53:59.526] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:00 [17:54:00.599] INFO: prover-client:proving-broker Proving Broker started 17:54:00 [17:54:00.607] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9022622c43ec8c86 settled=false {"provingJobId":"1:BASE_PARITY:9022622c43ec8c86","status":"pending"} 17:54:00 [17:54:00.611] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:95683d87dcd51a5f settled=false {"provingJobId":"1:BASE_PARITY:95683d87dcd51a5f","status":"pending"} 17:54:00 [17:54:00.613] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:9022622c43ec8c86 type=BASE_PARITY startedAt=1750096439547 older agent has taken job {"provingJobId":"1:BASE_PARITY:9022622c43ec8c86"} 17:54:00 [17:54:00.616] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9022622c43ec8c86 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:9022622c43ec8c86"} 17:54:00 [17:54:00.621] INFO: prover-client:proving-broker Proving Broker started 17:54:00 [17:54:00.622] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fb8efbe5ae67d25e epochNumber=1 {"provingJobId":"1:BASE_PARITY:fb8efbe5ae67d25e"} 17:54:00 [17:54:00.623] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwEPHV6/1 with map size 1073741824 17:54:00 [17:54:00.625] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:00 [17:54:00.680] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9a0c79ea85340235 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9a0c79ea85340235"} 17:54:00 [17:54:00.681] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwEPHV6/2 with map size 1073741824 17:54:00 [17:54:00.683] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:01 [17:54:01.710] INFO: prover-client:proving-broker Proving Broker started 17:54:01 [17:54:01.717] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fb8efbe5ae67d25e settled=false {"provingJobId":"1:BASE_PARITY:fb8efbe5ae67d25e","status":"pending"} 17:54:01 [17:54:01.719] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9a0c79ea85340235 settled=false {"provingJobId":"1:BASE_PARITY:9a0c79ea85340235","status":"pending"} 17:54:01 [17:54:01.719] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:fb8efbe5ae67d25e type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:fb8efbe5ae67d25e"} 17:54:01 [17:54:01.725] INFO: prover-client:proving-broker Proving Broker started 17:54:01 [17:54:01.726] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8d6aed8b19d0dc1b epochNumber=1 {"provingJobId":"1:BASE_PARITY:8d6aed8b19d0dc1b"} 17:54:01 [17:54:01.727] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlP0WWU/1 with map size 1073741824 17:54:01 [17:54:01.730] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:01 [17:54:01.751] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8b9f79bfb212ff2c epochNumber=2 {"provingJobId":"1:BASE_PARITY:8b9f79bfb212ff2c"} 17:54:01 [17:54:01.752] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlP0WWU/2 with map size 1073741824 17:54:01 [17:54:01.754] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:02 [17:54:02.774] INFO: prover-client:proving-broker Proving Broker started 17:54:02 [17:54:02.777] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8d6aed8b19d0dc1b settled=false {"provingJobId":"1:BASE_PARITY:8d6aed8b19d0dc1b","status":"pending"} 17:54:02 [17:54:02.779] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8b9f79bfb212ff2c settled=false {"provingJobId":"1:BASE_PARITY:8b9f79bfb212ff2c","status":"pending"} 17:54:02 [17:54:02.780] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8d6aed8b19d0dc1b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8d6aed8b19d0dc1b"} 17:54:02 [17:54:02.780] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8d6aed8b19d0dc1b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8d6aed8b19d0dc1b"} 17:54:02 [17:54:02.792] INFO: prover-client:proving-broker Proving Broker started 17:54:02 [17:54:02.793] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4e1a77bbbfdf87ab epochNumber=1 {"provingJobId":"1:BASE_PARITY:4e1a77bbbfdf87ab"} 17:54:02 [17:54:02.797] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test52TQiO/1 with map size 1073741824 17:54:02 [17:54:02.800] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:02 [17:54:02.824] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b0c33d049101b3fa epochNumber=2 {"provingJobId":"1:BASE_PARITY:b0c33d049101b3fa"} 17:54:02 [17:54:02.829] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test52TQiO/2 with map size 1073741824 17:54:02 [17:54:02.831] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:02 [17:54:02.850] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4e1a77bbbfdf87ab type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:4e1a77bbbfdf87ab"} 17:54:02 [17:54:02.857] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b0c33d049101b3fa type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b0c33d049101b3fa"} 17:54:02 [17:54:02.869] INFO: prover-client:proving-broker Proving Broker started 17:54:02 [17:54:02.870] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:12baa25c03f98663 epochNumber=1 {"provingJobId":"1:BASE_PARITY:12baa25c03f98663"} 17:54:02 [17:54:02.870] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnl5buZ/1 with map size 1073741824 17:54:02 [17:54:02.874] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:02 [17:54:02.894] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:afe4b62ee00f06a3 epochNumber=2 {"provingJobId":"1:BASE_PARITY:afe4b62ee00f06a3"} 17:54:02 [17:54:02.898] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnl5buZ/2 with map size 1073741824 17:54:02 [17:54:02.903] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:02 [17:54:02.921] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:12baa25c03f98663 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:12baa25c03f98663"} 17:54:02 [17:54:02.922] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:12baa25c03f98663 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:12baa25c03f98663"} 17:54:02 [17:54:02.930] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:afe4b62ee00f06a3 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:afe4b62ee00f06a3"} 17:54:02 [17:54:02.940] INFO: prover-client:proving-broker Proving Broker started 17:54:02 [17:54:02.941] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:22f7cabf84a47e52 err=test error {"provingJoId":"1:BASE_PARITY:22f7cabf84a47e52"} 17:54:02 [17:54:02.945] INFO: prover-client:proving-broker Proving Broker started 17:54:02 [17:54:02.946] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c5036bd0a57b2596 not found {"provingJobId":"1:BASE_PARITY:c5036bd0a57b2596"} 17:54:02 [17:54:02.948] INFO: prover-client:proving-broker Proving Broker started 17:54:02 [17:54:02.949] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:989459d82661bd4a epochNumber=1 {"provingJobId":"1:BASE_PARITY:989459d82661bd4a"} 17:54:02 [17:54:02.949] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcxjNVK/1 with map size 1073741824 17:54:02 [17:54:02.951] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:02 [17:54:02.977] INFO: prover-client:proving-broker Proving Broker started 17:54:02 [17:54:02.978] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:025362cc549fb0d0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:025362cc549fb0d0"} 17:54:02 [17:54:02.979] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsiboyf/1 with map size 1073741824 17:54:02 [17:54:02.981] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.107] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:025362cc549fb0d0 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:025362cc549fb0d0"} 17:54:03 [17:54:03.121] INFO: prover-client:proving-broker Proving Broker started 17:54:03 [17:54:03.122] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b12ea42239f0fcb5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b12ea42239f0fcb5"} 17:54:03 [17:54:03.122] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9EUtp7/1 with map size 1073741824 17:54:03 [17:54:03.124] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.248] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b12ea42239f0fcb5 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:b12ea42239f0fcb5"} 17:54:03 [17:54:03.251] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f43e1d2a9cc4dd5b epochNumber=10 {"provingJobId":"1:BASE_PARITY:f43e1d2a9cc4dd5b"} 17:54:03 [17:54:03.252] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-test9EUtp7/10 with map size 1073741824 17:54:03 [17:54:03.254] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.274] INFO: prover-client:proving-broker Cleaned up jobs=1 17:54:03 [17:54:03.275] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 17:54:03 [17:54:03.403] INFO: prover-client:proving-broker Proving Broker started 17:54:03 [17:54:03.404] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aa6a46400b99b50a epochNumber=1 {"provingJobId":"1:BASE_PARITY:aa6a46400b99b50a"} 17:54:03 [17:54:03.405] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testE2qQb1/1 with map size 1073741824 17:54:03 [17:54:03.407] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.607] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:aa6a46400b99b50a timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:aa6a46400b99b50a"} 17:54:03 [17:54:03.640] INFO: prover-client:proving-broker Proving Broker started 17:54:03 [17:54:03.641] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:57daee542ff5ab0f epochNumber=1 {"provingJobId":"1:BASE_PARITY:57daee542ff5ab0f"} 17:54:03 [17:54:03.643] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testN0S0b5/1 with map size 1073741824 17:54:03 [17:54:03.646] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.669] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:57daee542ff5ab0f type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:57daee542ff5ab0f"} 17:54:03 [17:54:03.676] INFO: prover-client:proving-broker Proving Broker started 17:54:03 [17:54:03.678] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4d3bf4606d7b9486 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4d3bf4606d7b9486"} 17:54:03 [17:54:03.679] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLB3lKA/1 with map size 1073741824 17:54:03 [17:54:03.681] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.711] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4d3bf4606d7b9486 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:4d3bf4606d7b9486"} 17:54:03 [17:54:03.712] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4d3bf4606d7b9486 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:4d3bf4606d7b9486"} 17:54:03 [17:54:03.721] INFO: prover-client:proving-broker Proving Broker started 17:54:03 [17:54:03.722] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3f4e7d7ef1a53c8b epochNumber=1 {"provingJobId":"1:BASE_PARITY:3f4e7d7ef1a53c8b"} 17:54:03 [17:54:03.724] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDy4Zcq/1 with map size 1073741824 17:54:03 [17:54:03.727] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.760] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3f4e7d7ef1a53c8b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3f4e7d7ef1a53c8b"} 17:54:03 [17:54:03.775] INFO: prover-client:proving-broker Proving Broker started 17:54:03 [17:54:03.776] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0f309d179e4ceeb9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0f309d179e4ceeb9"} 17:54:03 [17:54:03.777] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2axbiU/1 with map size 1073741824 17:54:03 [17:54:03.779] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.799] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0f309d179e4ceeb9 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:0f309d179e4ceeb9"} 17:54:03 [17:54:03.800] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c4b3e3f9d9d8b0a2 epochNumber=3 {"provingJobId":"1:BASE_PARITY:c4b3e3f9d9d8b0a2"} 17:54:03 [17:54:03.801] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test2axbiU/3 with map size 1073741824 17:54:03 [17:54:03.805] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.826] INFO: prover-client:proving-broker Cleaned up jobs=1 17:54:03 [17:54:03.827] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 17:54:03 [17:54:03.850] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0f309d179e4ceeb9 err=test error {"provingJoId":"1:BASE_PARITY:0f309d179e4ceeb9"} 17:54:03 [17:54:03.856] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKNnyjE/1 with map size 1073741824 17:54:03 [17:54:03.858] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.883] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKNnyjE/2 with map size 1073741824 17:54:03 [17:54:03.888] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.918] INFO: prover-client:proving-broker Proving Broker started 17:54:03 [17:54:03.920] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d6aab8cc53360b85 settled=false {"provingJobId":"1:BASE_PARITY:d6aab8cc53360b85","status":"pending"} 17:54:03 [17:54:03.922] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f141e1f039055417 settled=false {"provingJobId":"1:BASE_PARITY:f141e1f039055417","status":"pending"} 17:54:03 [17:54:03.931] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBDcS3V/1 with map size 1073741824 17:54:03 [17:54:03.933] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.958] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testBDcS3V/2 with map size 1073741824 17:54:03 [17:54:03.964] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:03 [17:54:03.997] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.001] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:541da4911e8e07da settled=true {"provingJobId":"1:BASE_PARITY:541da4911e8e07da","status":"fulfilled"} 17:54:04 [17:54:04.004] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:a88bfd0fc1db2cd0 settled=true {"provingJobId":"2:BASE_PARITY:a88bfd0fc1db2cd0","status":"fulfilled"} 17:54:04 [17:54:04.018] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNWT1TS/1 with map size 1073741824 17:54:04 [17:54:04.022] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.053] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNWT1TS/2 with map size 1073741824 17:54:04 [17:54:04.056] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.086] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.089] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:beb7e9966d019d2d settled=true {"provingJobId":"1:BASE_PARITY:beb7e9966d019d2d","status":"fulfilled"} 17:54:04 [17:54:04.092] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:47c023dd8c4e864c settled=false {"provingJobId":"1:BASE_PARITY:47c023dd8c4e864c","status":"pending"} 17:54:04 [17:54:04.103] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.104] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f1f1255bd26e7fad epochNumber=1 {"provingJobId":"1:BASE_PARITY:f1f1255bd26e7fad"} 17:54:04 [17:54:04.106] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test47unhe/1 with map size 1073741824 17:54:04 [17:54:04.112] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.146] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.147] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:28c254b508d6b53c epochNumber=1 {"provingJobId":"1:BASE_PARITY:28c254b508d6b53c"} 17:54:04 [17:54:04.147] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:28c254b508d6b53c: Error: db error: Error: db error 17:54:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:28c254b508d6b53c"} 17:54:04 [17:54:04.150] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.150] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2e0db1bc098b9379 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2e0db1bc098b9379"} 17:54:04 [17:54:04.154] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDGPHBk/1 with map size 1073741824 17:54:04 [17:54:04.157] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.183] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2e0db1bc098b9379 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:2e0db1bc098b9379"} 17:54:04 [17:54:04.183] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2e0db1bc098b9379 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2e0db1bc098b9379"} 17:54:04 [17:54:04.203] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.204] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eeb55ee700870e9c epochNumber=1 {"provingJobId":"1:BASE_PARITY:eeb55ee700870e9c"} 17:54:04 [17:54:04.207] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUs6Phq/1 with map size 1073741824 17:54:04 [17:54:04.216] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.239] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:eeb55ee700870e9c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:eeb55ee700870e9c"} 17:54:04 [17:54:04.240] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:eeb55ee700870e9c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:eeb55ee700870e9c"} 17:54:04 [17:54:04.241] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:eeb55ee700870e9c: Error: db error 17:54:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:eeb55ee700870e9c"} 17:54:04 [17:54:04.245] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.254] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b6a5a20c54c0e13e epochNumber=1 {"provingJobId":"1:BASE_PARITY:b6a5a20c54c0e13e"} 17:54:04 [17:54:04.257] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdXdvUO/1 with map size 1073741824 17:54:04 [17:54:04.262] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.284] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b6a5a20c54c0e13e type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b6a5a20c54c0e13e"} 17:54:04 [17:54:04.284] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b6a5a20c54c0e13e type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b6a5a20c54c0e13e"} 17:54:04 [17:54:04.300] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.301] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0a802b44b725894b epochNumber=1 {"provingJobId":"1:BASE_PARITY:0a802b44b725894b"} 17:54:04 [17:54:04.302] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testC44uxY/1 with map size 1073741824 17:54:04 [17:54:04.305] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.327] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0a802b44b725894b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0a802b44b725894b"} 17:54:04 [17:54:04.328] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0a802b44b725894b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:0a802b44b725894b"} 17:54:04 [17:54:04.329] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:0a802b44b725894b jobErr=test error: Error: db error 17:54:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:0a802b44b725894b"} 17:54:04 [17:54:04.338] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.339] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8f738b4fe03b7cc8 not found {"provingJobId":"1:BASE_PARITY:8f738b4fe03b7cc8"} 17:54:04 [17:54:04.345] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.346] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:eb15319af0578875 err=test error {"provingJoId":"1:BASE_PARITY:eb15319af0578875"} 17:54:04 [17:54:04.353] INFO: prover-client:proving-broker Proving Broker started 17:54:04 [17:54:04.364] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8d4aa20a556f68f3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8d4aa20a556f68f3"} 17:54:04 [17:54:04.365] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQy8OCZ/1 with map size 1073741824 17:54:04 [17:54:04.367] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.391] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8d4aa20a556f68f3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8d4aa20a556f68f3"} 17:54:04 [17:54:04.392] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8d4aa20a556f68f3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8d4aa20a556f68f3"} 17:54:04 [17:54:04.413] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:3a67d4d69f61bd02 epochNumber=2 {"provingJobId":"2:BASE_PARITY:3a67d4d69f61bd02"} 17:54:04 [17:54:04.415] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQy8OCZ/2 with map size 1073741824 17:54:04 [17:54:04.421] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.445] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:3a67d4d69f61bd02 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:3a67d4d69f61bd02"} 17:54:04 [17:54:04.446] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:3a67d4d69f61bd02 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:3a67d4d69f61bd02"} 17:54:04 [17:54:04.515] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:23b99976dfe9c5cb epochNumber=3 {"provingJobId":"3:BASE_PARITY:23b99976dfe9c5cb"} 17:54:04 [17:54:04.517] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testQy8OCZ/3 with map size 1073741824 17:54:04 [17:54:04.520] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.557] INFO: prover-client:proving-broker Cleaned up jobs=1 17:54:04 [17:54:04.558] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 17:54:04 [17:54:04.618] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:23254c9031a53d5a epochNumber=4 {"provingJobId":"4:BASE_PARITY:23254c9031a53d5a"} 17:54:04 [17:54:04.621] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testQy8OCZ/4 with map size 1073741824 17:54:04 [17:54:04.631] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.664] INFO: prover-client:proving-broker Cleaned up jobs=1 17:54:04 [17:54:04.664] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 17:54:04 [17:54:04.726] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:01b0028fb4d1b2f8 epochNumber=5 {"provingJobId":"5:BASE_PARITY:01b0028fb4d1b2f8"} 17:54:04 [17:54:04.730] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testQy8OCZ/5 with map size 1073741824 17:54:04 [17:54:04.740] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 17:54:04 [17:54:04.795] INFO: prover-client:proving-broker Cleaned up jobs=1 17:54:04 [17:54:04.796] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 17:54:04 PASS src/proving_broker/proving_broker.test.ts (13.544 s) 17:54:04 ProvingBroker 17:54:04 Producer API 17:54:04 ✓ refuses stale jobs (60 ms) 17:54:04 ✓ enqueues jobs (4 ms) 17:54:04 ✓ ignores duplicate jobs (3 ms) 17:54:04 ✓ reports correct status when enqueuing repeat jobs (3 ms) 17:54:04 ✓ throws an error in case of duplicate job IDs (21 ms) 17:54:04 ✓ returns not-found status for non-existing jobs (3 ms) 17:54:04 ✓ cancels jobs in queue (6 ms) 17:54:04 ✓ cancels jobs in-progress (4 ms) 17:54:04 ✓ returns job result if successful (6 ms) 17:54:04 ✓ returns job error if failed (3 ms) 17:54:04 ✓ correctly returns job status for concurrent writes (3 ms) 17:54:04 ✓ refuses stale jobs (42 ms) 17:54:04 ✓ enqueues jobs (38 ms) 17:54:04 ✓ ignores duplicate jobs (30 ms) 17:54:04 ✓ reports correct status when enqueuing repeat jobs (35 ms) 17:54:04 ✓ throws an error in case of duplicate job IDs (42 ms) 17:54:04 ✓ returns not-found status for non-existing jobs (3 ms) 17:54:04 ✓ cancels jobs in queue (32 ms) 17:54:04 ✓ cancels jobs in-progress (36 ms) 17:54:04 ✓ returns job result if successful (34 ms) 17:54:04 ✓ returns job error if failed (33 ms) 17:54:04 ✓ correctly returns job status for concurrent writes (49 ms) 17:54:04 Consumer API 17:54:04 ✓ returns undefined if no jobs are available 17:54:04 ✓ returns jobs in priority order (1 ms) 17:54:04 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 17:54:04 ✓ returns a job if it is in the allowList (2 ms) 17:54:04 ✓ returns the most important job if it is in the allowList (2 ms) 17:54:04 ✓ returns any job if filter is empty (2 ms) 17:54:04 ✓ returns a new job when reporting job success (2 ms) 17:54:04 ✓ returns a new job when reporting permanent error (1 ms) 17:54:04 ✓ returns a new job when reporting retry-able error (1 ms) 17:54:04 ✓ returns a new job when reporting progress if current one is cancelled (11 ms) 17:54:04 ✓ returns a new job if job is already in progress elsewhere (1058 ms) 17:54:04 ✓ avoids sending the same job to a new agent after a restart (1005 ms) 17:54:04 ✓ avoids sending a completed job to a new agent after a restart (1008 ms) 17:54:04 ✓ tracks job result if in progress (5 ms) 17:54:04 ✓ tracks job result even if job is in queue (3 ms) 17:54:04 ✓ ignores reported job error if unknown job (2 ms) 17:54:04 ✓ ignores job result if unknown job (1 ms) 17:54:04 ✓ returns undefined if no jobs are available (3 ms) 17:54:04 ✓ returns jobs in priority order (49 ms) 17:54:04 ✓ returns undefined if no jobs are available for the given allowList (32 ms) 17:54:04 ✓ returns a job if it is in the allowList (76 ms) 17:54:04 ✓ returns the most important job if it is in the allowList (65 ms) 17:54:04 ✓ returns any job if filter is empty (62 ms) 17:54:04 ✓ returns a new job when reporting job success (44 ms) 17:54:04 ✓ returns a new job when reporting permanent error (42 ms) 17:54:04 ✓ returns a new job when reporting retry-able error (39 ms) 17:54:04 ✓ returns a new job when reporting progress if current one is cancelled (39 ms) 17:54:04 ✓ returns a new job if job is already in progress elsewhere (1133 ms) 17:54:04 ✓ avoids sending the same job to a new agent after a restart (1103 ms) 17:54:04 ✓ avoids sending a completed job to a new agent after a restart (1066 ms) 17:54:04 ✓ tracks job result if in progress (77 ms) 17:54:04 ✓ tracks job result even if job is in queue (71 ms) 17:54:04 ✓ ignores reported job error if unknown job (3 ms) 17:54:04 ✓ ignores job result if unknown job (4 ms) 17:54:04 Timeouts 17:54:04 ✓ tracks in progress jobs (8 ms) 17:54:04 ✓ re-enqueues jobs that time out (109 ms) 17:54:04 ✓ cancel stale jobs that time out (235 ms) 17:54:04 ✓ keeps the jobs in progress while it is alive (206 ms) 17:54:04 ✓ tracks in progress jobs (25 ms) 17:54:04 ✓ re-enqueues jobs that time out (144 ms) 17:54:04 ✓ cancel stale jobs that time out (283 ms) 17:54:04 ✓ keeps the jobs in progress while it is alive (233 ms) 17:54:04 Retries 17:54:04 ✓ retries jobs (9 ms) 17:54:04 ✓ retries up to a maximum number of times (4 ms) 17:54:04 ✓ passing retry=false does not retry (7 ms) 17:54:04 ✓ does not retry if job is stale (30 ms) 17:54:04 ✓ retries jobs (36 ms) 17:54:04 ✓ retries up to a maximum number of times (47 ms) 17:54:04 ✓ passing retry=false does not retry (54 ms) 17:54:04 ✓ does not retry if job is stale (78 ms) 17:54:04 Database management 17:54:04 ✓ re-enqueues proof requests on start (5 ms) 17:54:04 ✓ restores proof results on start (3 ms) 17:54:04 ✓ only re-enqueues unfinished jobs (4 ms) 17:54:04 ✓ saves job when enqueued (5 ms) 17:54:04 ✓ does not retain job if database fails to save (4 ms) 17:54:04 ✓ saves job result (3 ms) 17:54:04 ✓ saves result even if database fails to save (3 ms) 17:54:04 ✓ saves job error (3 ms) 17:54:04 ✓ saves job error even if database fails to save (3 ms) 17:54:04 ✓ does not save job result if job is unknown (2 ms) 17:54:04 ✓ does not save job error if job is unknown (1 ms) 17:54:04 ✓ cleans up old jobs periodically (264 ms) 17:54:04 ✓ re-enqueues proof requests on start (73 ms) 17:54:04 ✓ restores proof results on start (85 ms) 17:54:04 ✓ only re-enqueues unfinished jobs (84 ms) 17:54:04 ✓ saves job when enqueued (47 ms) 17:54:04 ✓ does not retain job if database fails to save (5 ms) 17:54:04 ✓ saves job result (43 ms) 17:54:04 ✓ saves result even if database fails to save (50 ms) 17:54:04 ✓ saves job error (52 ms) 17:54:04 ✓ saves job error even if database fails to save (36 ms) 17:54:04 ✓ does not save job result if job is unknown (8 ms) 17:54:04 ✓ does not save job error if job is unknown (6 ms) 17:54:04 ✓ cleans up old jobs periodically (473 ms) 17:54:04 17:54:04 Test Suites: 1 passed, 1 total 17:54:04 Tests: 96 passed, 96 total 17:54:04 Snapshots: 0 total 17:54:04 Time: 13.652 s 17:54:04 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 17:54:04 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?