Parent Log: http://ci.aztec-labs.com/4b4bb6a11b9c4e41 Command: 98619016930b559c yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/9b15672dd871e73616eaa70188d438d5a1e5f970 Env: REF_NAME=sb/doc-bigfield CURRENT_VERSION=0.87.6 CI_FULL=0 Date: Thu Jun 19 19:38:15 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=sb_doc-bigfield_amd64 Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 19:38:19 [19:38:19.568] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.576] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:13b38494693cebcb epochNumber=42 {"provingJobId":"1:BASE_PARITY:13b38494693cebcb"} 19:38:19 [19:38:19.579] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:8e040c2dbe3251da epochNumber=1 {"provingJobId":"1:BASE_PARITY:8e040c2dbe3251da"} 19:38:19 [19:38:19.615] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.616] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:88d4567f353c26f0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:88d4567f353c26f0"} 19:38:19 [19:38:19.618] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c0e1c4c4c4328be8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c0e1c4c4c4328be8"} 19:38:19 [19:38:19.620] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.620] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:660a09861545b1f6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:660a09861545b1f6"} 19:38:19 [19:38:19.622] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:660a09861545b1f6 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:660a09861545b1f6"} 19:38:19 [19:38:19.624] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.625] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f09001b0e38c2d79 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f09001b0e38c2d79"} 19:38:19 [19:38:19.626] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f09001b0e38c2d79 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f09001b0e38c2d79"} 19:38:19 [19:38:19.627] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f09001b0e38c2d79 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f09001b0e38c2d79"} 19:38:19 [19:38:19.627] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f09001b0e38c2d79 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f09001b0e38c2d79"} 19:38:19 [19:38:19.629] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.629] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3583d20920585481 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3583d20920585481"} 19:38:19 [19:38:19.662] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.663] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.664] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cf131d088fad9904 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cf131d088fad9904"} 19:38:19 [19:38:19.664] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:cf131d088fad9904 {"provingJobId":"1:BASE_PARITY:cf131d088fad9904"} 19:38:19 [19:38:19.665] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cf131d088fad9904 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cf131d088fad9904"} 19:38:19 [19:38:19.665] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cf131d088fad9904 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:cf131d088fad9904"} 19:38:19 [19:38:19.667] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.667] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7629b183bbc75ac5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7629b183bbc75ac5"} 19:38:19 [19:38:19.668] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7629b183bbc75ac5 {"provingJobId":"1:BASE_PARITY:7629b183bbc75ac5"} 19:38:19 [19:38:19.668] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7629b183bbc75ac5 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:7629b183bbc75ac5"} 19:38:19 [19:38:19.670] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.671] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3d3ce1d70bc93d24 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3d3ce1d70bc93d24"} 19:38:19 [19:38:19.671] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3d3ce1d70bc93d24 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3d3ce1d70bc93d24"} 19:38:19 [19:38:19.672] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3d3ce1d70bc93d24 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3d3ce1d70bc93d24"} 19:38:19 [19:38:19.673] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.674] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c1747632c4ac456d epochNumber=1 {"provingJobId":"1:BASE_PARITY:c1747632c4ac456d"} 19:38:19 [19:38:19.674] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c1747632c4ac456d type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c1747632c4ac456d"} 19:38:19 [19:38:19.674] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c1747632c4ac456d type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:c1747632c4ac456d"} 19:38:19 [19:38:19.676] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.676] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:abfa62e66b721c60 epochNumber=0 {"provingJobId":"1:BASE_PARITY:abfa62e66b721c60"} 19:38:19 [19:38:19.676] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:abfa62e66b721c60 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:abfa62e66b721c60"} 19:38:19 [19:38:19.677] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7fcd35802cdc4647 epochNumber=0 {"provingJobId":"1:BASE_PARITY:7fcd35802cdc4647"} 19:38:19 [19:38:19.677] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:abfa62e66b721c60 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:abfa62e66b721c60"} 19:38:19 [19:38:19.677] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c511556b1a2e9e7b epochNumber=0 {"provingJobId":"1:BASE_PARITY:c511556b1a2e9e7b"} 19:38:19 [19:38:19.679] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.680] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.681] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ff0cc13010c7b256 epochNumber=2 {"provingJobId":"1:BASE_PARITY:ff0cc13010c7b256"} 19:38:19 [19:38:19.681] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5177347cc5561828 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5177347cc5561828"} 19:38:19 [19:38:19.682] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.683] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:098e07d51beb140b epochNumber=1 {"provingJobId":"1:BASE_PARITY:098e07d51beb140b"} 19:38:19 [19:38:19.684] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.684] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7d96eed027b20e8f epochNumber=1 {"provingJobId":"1:BASE_PARITY:7d96eed027b20e8f"} 19:38:19 [19:38:19.685] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4f1839f7e09ddc09 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4f1839f7e09ddc09"} 19:38:19 [19:38:19.685] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:35ced3615059c8b2 epochNumber=2 {"provingJobId":"1:BASE_PARITY:35ced3615059c8b2"} 19:38:19 [19:38:19.685] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8a5b8ac75bcad9f6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8a5b8ac75bcad9f6"} 19:38:19 [19:38:19.686] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.687] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8f193cf33f6dc825 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8f193cf33f6dc825"} 19:38:19 [19:38:19.687] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:004fee53ea8f6e65 epochNumber=1 {"provingJobId":"1:BASE_PARITY:004fee53ea8f6e65"} 19:38:19 [19:38:19.688] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fc948442c94a5672 epochNumber=2 {"provingJobId":"1:BASE_PARITY:fc948442c94a5672"} 19:38:19 [19:38:19.688] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:77880b36ab6d52f9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:77880b36ab6d52f9"} 19:38:19 [19:38:19.689] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.690] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e8af301e92fde1b5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e8af301e92fde1b5"} 19:38:19 [19:38:19.690] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0581ff716c6f6125 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0581ff716c6f6125"} 19:38:19 [19:38:19.690] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1a618c7653cc872f epochNumber=2 {"provingJobId":"1:BASE_PARITY:1a618c7653cc872f"} 19:38:19 [19:38:19.691] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d4d995f97801adf9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d4d995f97801adf9"} 19:38:19 [19:38:19.692] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.692] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7f603d4ff0c20f89 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7f603d4ff0c20f89"} 19:38:19 [19:38:19.693] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bf989fcf1be1f2bc epochNumber=1 {"provingJobId":"1:BASE_PARITY:bf989fcf1be1f2bc"} 19:38:19 [19:38:19.693] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7f603d4ff0c20f89 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:7f603d4ff0c20f89"} 19:38:19 [19:38:19.694] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.695] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9c03df553bf26573 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9c03df553bf26573"} 19:38:19 [19:38:19.696] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4e36fa6525e6a8a0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4e36fa6525e6a8a0"} 19:38:19 [19:38:19.696] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9c03df553bf26573 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:9c03df553bf26573"} 19:38:19 [19:38:19.697] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.697] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b831ec64b01d2cb1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b831ec64b01d2cb1"} 19:38:19 [19:38:19.698] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d1b45369d637862a epochNumber=1 {"provingJobId":"1:BASE_PARITY:d1b45369d637862a"} 19:38:19 [19:38:19.698] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b831ec64b01d2cb1 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:b831ec64b01d2cb1"} 19:38:19 [19:38:19.699] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.700] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a56017c99b8cb85b epochNumber=1 {"provingJobId":"1:BASE_PARITY:a56017c99b8cb85b"} 19:38:19 [19:38:19.700] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:a56017c99b8cb85b {"provingJobId":"1:BASE_PARITY:a56017c99b8cb85b"} 19:38:19 [19:38:19.701] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a56017c99b8cb85b type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:a56017c99b8cb85b"} 19:38:19 [19:38:19.701] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a452325d5defbe4a epochNumber=1 {"provingJobId":"1:BASE_PARITY:a452325d5defbe4a"} 19:38:19 [19:38:19.702] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a56017c99b8cb85b has already been completed {"provingJobId":"1:BASE_PARITY:a56017c99b8cb85b"} 19:38:19 [19:38:19.703] INFO: prover-client:proving-broker Proving Broker started 19:38:19 [19:38:19.703] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1a3f26cf2bb2068b epochNumber=1 {"provingJobId":"1:BASE_PARITY:1a3f26cf2bb2068b"} 19:38:19 [19:38:19.704] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fcd39a0e59b3683a epochNumber=2 {"provingJobId":"1:BASE_PARITY:fcd39a0e59b3683a"} 19:38:20 [19:38:20.757] INFO: prover-client:proving-broker Proving Broker started 19:38:20 [19:38:20.757] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1a3f26cf2bb2068b settled=false {"provingJobId":"1:BASE_PARITY:1a3f26cf2bb2068b","status":"pending"} 19:38:20 [19:38:20.758] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fcd39a0e59b3683a settled=false {"provingJobId":"1:BASE_PARITY:fcd39a0e59b3683a","status":"pending"} 19:38:20 [19:38:20.758] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:1a3f26cf2bb2068b type=BASE_PARITY startedAt=1750361899704 older agent has taken job {"provingJobId":"1:BASE_PARITY:1a3f26cf2bb2068b"} 19:38:20 [19:38:20.759] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1a3f26cf2bb2068b type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:1a3f26cf2bb2068b"} 19:38:20 [19:38:20.760] INFO: prover-client:proving-broker Proving Broker started 19:38:20 [19:38:20.760] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7997a9e297c27d4e epochNumber=1 {"provingJobId":"1:BASE_PARITY:7997a9e297c27d4e"} 19:38:20 [19:38:20.760] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:56640c94eebdd12f epochNumber=2 {"provingJobId":"1:BASE_PARITY:56640c94eebdd12f"} 19:38:21 [19:38:21.761] INFO: prover-client:proving-broker Proving Broker started 19:38:21 [19:38:21.762] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7997a9e297c27d4e settled=false {"provingJobId":"1:BASE_PARITY:7997a9e297c27d4e","status":"pending"} 19:38:21 [19:38:21.762] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:56640c94eebdd12f settled=false {"provingJobId":"1:BASE_PARITY:56640c94eebdd12f","status":"pending"} 19:38:21 [19:38:21.763] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7997a9e297c27d4e type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:7997a9e297c27d4e"} 19:38:21 [19:38:21.764] INFO: prover-client:proving-broker Proving Broker started 19:38:21 [19:38:21.765] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5ab50451204cb5f3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5ab50451204cb5f3"} 19:38:21 [19:38:21.765] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f966d28e53391949 epochNumber=2 {"provingJobId":"1:BASE_PARITY:f966d28e53391949"} 19:38:22 [19:38:22.766] INFO: prover-client:proving-broker Proving Broker started 19:38:22 [19:38:22.767] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5ab50451204cb5f3 settled=false {"provingJobId":"1:BASE_PARITY:5ab50451204cb5f3","status":"pending"} 19:38:22 [19:38:22.767] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f966d28e53391949 settled=false {"provingJobId":"1:BASE_PARITY:f966d28e53391949","status":"pending"} 19:38:22 [19:38:22.768] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5ab50451204cb5f3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5ab50451204cb5f3"} 19:38:22 [19:38:22.768] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5ab50451204cb5f3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5ab50451204cb5f3"} 19:38:22 [19:38:22.770] INFO: prover-client:proving-broker Proving Broker started 19:38:22 [19:38:22.771] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c4179fbd90004305 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c4179fbd90004305"} 19:38:22 [19:38:22.771] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:76bf0fe100dcde9f epochNumber=2 {"provingJobId":"1:BASE_PARITY:76bf0fe100dcde9f"} 19:38:22 [19:38:22.772] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c4179fbd90004305 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:c4179fbd90004305"} 19:38:22 [19:38:22.772] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:76bf0fe100dcde9f type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:76bf0fe100dcde9f"} 19:38:22 [19:38:22.774] INFO: prover-client:proving-broker Proving Broker started 19:38:22 [19:38:22.774] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ecbc0408f4bdc4d0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ecbc0408f4bdc4d0"} 19:38:22 [19:38:22.775] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7e51d5c6c812cfd1 epochNumber=2 {"provingJobId":"1:BASE_PARITY:7e51d5c6c812cfd1"} 19:38:22 [19:38:22.775] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ecbc0408f4bdc4d0 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ecbc0408f4bdc4d0"} 19:38:22 [19:38:22.775] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ecbc0408f4bdc4d0 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ecbc0408f4bdc4d0"} 19:38:22 [19:38:22.775] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7e51d5c6c812cfd1 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:7e51d5c6c812cfd1"} 19:38:22 [19:38:22.776] INFO: prover-client:proving-broker Proving Broker started 19:38:22 [19:38:22.777] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1d763941c5a7f55f err=test error {"provingJoId":"1:BASE_PARITY:1d763941c5a7f55f"} 19:38:22 [19:38:22.778] INFO: prover-client:proving-broker Proving Broker started 19:38:22 [19:38:22.778] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ef05a088a54c039b not found {"provingJobId":"1:BASE_PARITY:ef05a088a54c039b"} 19:38:22 [19:38:22.780] INFO: prover-client:proving-broker Proving Broker started 19:38:22 [19:38:22.780] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a8b0337a738d6aa4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a8b0337a738d6aa4"} 19:38:22 [19:38:22.781] INFO: prover-client:proving-broker Proving Broker started 19:38:22 [19:38:22.782] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3eb0522c64d1683a epochNumber=1 {"provingJobId":"1:BASE_PARITY:3eb0522c64d1683a"} 19:38:22 [19:38:22.884] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3eb0522c64d1683a timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:3eb0522c64d1683a"} 19:38:22 [19:38:22.892] INFO: prover-client:proving-broker Proving Broker started 19:38:22 [19:38:22.892] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:be6825fefe56e929 epochNumber=1 {"provingJobId":"1:BASE_PARITY:be6825fefe56e929"} 19:38:22 [19:38:22.993] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:be6825fefe56e929 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:be6825fefe56e929"} 19:38:22 [19:38:22.995] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:765e7dca5136c16f epochNumber=10 {"provingJobId":"1:BASE_PARITY:765e7dca5136c16f"} 19:38:23 [19:38:23.020] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:23 [19:38:23.121] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.122] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5a6a041beadf093f epochNumber=1 {"provingJobId":"1:BASE_PARITY:5a6a041beadf093f"} 19:38:23 [19:38:23.273] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5a6a041beadf093f timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:5a6a041beadf093f"} 19:38:23 [19:38:23.324] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.325] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:39c1f6e9e2066996 epochNumber=1 {"provingJobId":"1:BASE_PARITY:39c1f6e9e2066996"} 19:38:23 [19:38:23.326] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:39c1f6e9e2066996 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:39c1f6e9e2066996"} 19:38:23 [19:38:23.328] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.328] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:db053fe0435fa3af epochNumber=1 {"provingJobId":"1:BASE_PARITY:db053fe0435fa3af"} 19:38:23 [19:38:23.329] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:db053fe0435fa3af type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:db053fe0435fa3af"} 19:38:23 [19:38:23.330] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:db053fe0435fa3af type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:db053fe0435fa3af"} 19:38:23 [19:38:23.331] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.331] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4261c16ae749d8c7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4261c16ae749d8c7"} 19:38:23 [19:38:23.332] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4261c16ae749d8c7 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4261c16ae749d8c7"} 19:38:23 [19:38:23.333] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.334] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4e72271781ab2e31 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4e72271781ab2e31"} 19:38:23 [19:38:23.334] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4e72271781ab2e31 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:4e72271781ab2e31"} 19:38:23 [19:38:23.335] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bf5ce1a3e449bfdb epochNumber=3 {"provingJobId":"1:BASE_PARITY:bf5ce1a3e449bfdb"} 19:38:23 [19:38:23.359] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:23 [19:38:23.361] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:4e72271781ab2e31 err=test error {"provingJoId":"1:BASE_PARITY:4e72271781ab2e31"} 19:38:23 [19:38:23.363] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.363] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4e2b077931181c24 settled=false {"provingJobId":"1:BASE_PARITY:4e2b077931181c24","status":"pending"} 19:38:23 [19:38:23.363] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2fddb848993e44d5 settled=false {"provingJobId":"1:BASE_PARITY:2fddb848993e44d5","status":"pending"} 19:38:23 [19:38:23.366] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.366] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:369a092d90435c4f settled=true {"provingJobId":"1:BASE_PARITY:369a092d90435c4f","status":"fulfilled"} 19:38:23 [19:38:23.366] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:8f27f5e7abf6d92f settled=true {"provingJobId":"2:BASE_PARITY:8f27f5e7abf6d92f","status":"fulfilled"} 19:38:23 [19:38:23.367] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.368] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:221ce75384d47454 settled=true {"provingJobId":"1:BASE_PARITY:221ce75384d47454","status":"fulfilled"} 19:38:23 [19:38:23.368] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9ea3fe83b20d1f42 settled=false {"provingJobId":"1:BASE_PARITY:9ea3fe83b20d1f42","status":"pending"} 19:38:23 [19:38:23.369] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.370] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c1fa139b005c1455 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c1fa139b005c1455"} 19:38:23 [19:38:23.372] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.372] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:36cbc2e73c60e6be epochNumber=1 {"provingJobId":"1:BASE_PARITY:36cbc2e73c60e6be"} 19:38:23 [19:38:23.373] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:36cbc2e73c60e6be: Error: db error: Error: db error 19:38:23 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:36cbc2e73c60e6be"} 19:38:23 [19:38:23.374] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.374] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3ba215f23e2dac3f epochNumber=1 {"provingJobId":"1:BASE_PARITY:3ba215f23e2dac3f"} 19:38:23 [19:38:23.375] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3ba215f23e2dac3f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3ba215f23e2dac3f"} 19:38:23 [19:38:23.375] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3ba215f23e2dac3f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3ba215f23e2dac3f"} 19:38:23 [19:38:23.376] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.376] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d13c38cbb769213d epochNumber=1 {"provingJobId":"1:BASE_PARITY:d13c38cbb769213d"} 19:38:23 [19:38:23.377] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d13c38cbb769213d type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d13c38cbb769213d"} 19:38:23 [19:38:23.377] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d13c38cbb769213d type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d13c38cbb769213d"} 19:38:23 [19:38:23.377] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:d13c38cbb769213d: Error: db error 19:38:23 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:d13c38cbb769213d"} 19:38:23 [19:38:23.378] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.379] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ea3fa7f6cdbe4ebc epochNumber=1 {"provingJobId":"1:BASE_PARITY:ea3fa7f6cdbe4ebc"} 19:38:23 [19:38:23.379] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ea3fa7f6cdbe4ebc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ea3fa7f6cdbe4ebc"} 19:38:23 [19:38:23.379] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ea3fa7f6cdbe4ebc type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ea3fa7f6cdbe4ebc"} 19:38:23 [19:38:23.381] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.381] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ee84f3e2a30855c3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ee84f3e2a30855c3"} 19:38:23 [19:38:23.381] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ee84f3e2a30855c3 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ee84f3e2a30855c3"} 19:38:23 [19:38:23.381] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ee84f3e2a30855c3 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ee84f3e2a30855c3"} 19:38:23 [19:38:23.382] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:ee84f3e2a30855c3 jobErr=test error: Error: db error 19:38:23 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:ee84f3e2a30855c3"} 19:38:23 [19:38:23.383] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.383] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1b19397d94ef49e1 not found {"provingJobId":"1:BASE_PARITY:1b19397d94ef49e1"} 19:38:23 [19:38:23.384] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.385] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6cbb2a83ee6ff958 err=test error {"provingJoId":"1:BASE_PARITY:6cbb2a83ee6ff958"} 19:38:23 [19:38:23.386] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.396] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:45235839d0a3a32a epochNumber=1 {"provingJobId":"1:BASE_PARITY:45235839d0a3a32a"} 19:38:23 [19:38:23.396] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:45235839d0a3a32a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:45235839d0a3a32a"} 19:38:23 [19:38:23.397] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:45235839d0a3a32a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:45235839d0a3a32a"} 19:38:23 [19:38:23.407] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:2a90ccc204300e2e epochNumber=2 {"provingJobId":"2:BASE_PARITY:2a90ccc204300e2e"} 19:38:23 [19:38:23.407] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:2a90ccc204300e2e type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:2a90ccc204300e2e"} 19:38:23 [19:38:23.408] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:2a90ccc204300e2e type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:2a90ccc204300e2e"} 19:38:23 [19:38:23.468] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:5f155b9936480811 epochNumber=3 {"provingJobId":"3:BASE_PARITY:5f155b9936480811"} 19:38:23 [19:38:23.487] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:23 [19:38:23.530] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:977085533060f6f9 epochNumber=4 {"provingJobId":"4:BASE_PARITY:977085533060f6f9"} 19:38:23 [19:38:23.537] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:23 [19:38:23.593] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:af86882af6ab74b1 epochNumber=5 {"provingJobId":"5:BASE_PARITY:af86882af6ab74b1"} 19:38:23 [19:38:23.612] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:23 [19:38:23.647] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.648] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f07e023c3f76065f epochNumber=42 {"provingJobId":"1:BASE_PARITY:f07e023c3f76065f"} 19:38:23 [19:38:23.648] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testOO32LO/42 with map size 1073741824 19:38:23 [19:38:23.652] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.677] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:1bbecd78fa78428c epochNumber=1 {"provingJobId":"1:BASE_PARITY:1bbecd78fa78428c"} 19:38:23 [19:38:23.681] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.682] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b1a6a2c50a23512d epochNumber=1 {"provingJobId":"1:BASE_PARITY:b1a6a2c50a23512d"} 19:38:23 [19:38:23.683] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testo7uDjk/1 with map size 1073741824 19:38:23 [19:38:23.684] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.705] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a53d6e3efc0b8e98 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a53d6e3efc0b8e98"} 19:38:23 [19:38:23.714] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.715] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a97223e9c2d68945 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a97223e9c2d68945"} 19:38:23 [19:38:23.715] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9rcgQL/1 with map size 1073741824 19:38:23 [19:38:23.717] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.734] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:a97223e9c2d68945 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:a97223e9c2d68945"} 19:38:23 [19:38:23.737] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.738] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3895f8645012bcbb epochNumber=1 {"provingJobId":"1:BASE_PARITY:3895f8645012bcbb"} 19:38:23 [19:38:23.738] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvky1L7/1 with map size 1073741824 19:38:23 [19:38:23.740] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.758] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3895f8645012bcbb epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:3895f8645012bcbb"} 19:38:23 [19:38:23.759] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3895f8645012bcbb type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3895f8645012bcbb"} 19:38:23 [19:38:23.764] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3895f8645012bcbb epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:3895f8645012bcbb"} 19:38:23 [19:38:23.767] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.767] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:580efca4ecb170fc epochNumber=1 {"provingJobId":"1:BASE_PARITY:580efca4ecb170fc"} 19:38:23 [19:38:23.768] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGAh3i1/1 with map size 1073741824 19:38:23 [19:38:23.770] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.789] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.791] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.792] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bc252ded691c30dc epochNumber=1 {"provingJobId":"1:BASE_PARITY:bc252ded691c30dc"} 19:38:23 [19:38:23.793] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjFaAk5/1 with map size 1073741824 19:38:23 [19:38:23.796] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.813] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:bc252ded691c30dc {"provingJobId":"1:BASE_PARITY:bc252ded691c30dc"} 19:38:23 [19:38:23.814] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bc252ded691c30dc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:bc252ded691c30dc"} 19:38:23 [19:38:23.814] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bc252ded691c30dc type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:bc252ded691c30dc"} 19:38:23 [19:38:23.822] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.823] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:871b2837856799f9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:871b2837856799f9"} 19:38:23 [19:38:23.823] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbful4P/1 with map size 1073741824 19:38:23 [19:38:23.825] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.843] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:871b2837856799f9 {"provingJobId":"1:BASE_PARITY:871b2837856799f9"} 19:38:23 [19:38:23.844] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:871b2837856799f9 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:871b2837856799f9"} 19:38:23 [19:38:23.851] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.852] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cc3c5ca1fa3c5bcd epochNumber=1 {"provingJobId":"1:BASE_PARITY:cc3c5ca1fa3c5bcd"} 19:38:23 [19:38:23.853] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0Ut8zR/1 with map size 1073741824 19:38:23 [19:38:23.854] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.870] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cc3c5ca1fa3c5bcd type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cc3c5ca1fa3c5bcd"} 19:38:23 [19:38:23.871] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cc3c5ca1fa3c5bcd type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cc3c5ca1fa3c5bcd"} 19:38:23 [19:38:23.878] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.878] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:388fc1dfb7cde378 epochNumber=1 {"provingJobId":"1:BASE_PARITY:388fc1dfb7cde378"} 19:38:23 [19:38:23.879] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhyuxwV/1 with map size 1073741824 19:38:23 [19:38:23.880] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.896] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:388fc1dfb7cde378 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:388fc1dfb7cde378"} 19:38:23 [19:38:23.896] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:388fc1dfb7cde378 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:388fc1dfb7cde378"} 19:38:23 [19:38:23.903] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.903] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8285127562aefc81 epochNumber=0 {"provingJobId":"1:BASE_PARITY:8285127562aefc81"} 19:38:23 [19:38:23.904] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test3fqldK/0 with map size 1073741824 19:38:23 [19:38:23.905] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.923] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8285127562aefc81 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8285127562aefc81"} 19:38:23 [19:38:23.924] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ba1d346d7e01e128 epochNumber=0 {"provingJobId":"1:BASE_PARITY:ba1d346d7e01e128"} 19:38:23 [19:38:23.924] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:8285127562aefc81 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:8285127562aefc81"} 19:38:23 [19:38:23.924] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:adad656369b1b536 epochNumber=0 {"provingJobId":"1:BASE_PARITY:adad656369b1b536"} 19:38:23 [19:38:23.938] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.940] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.940] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:12f252fdc7480992 epochNumber=2 {"provingJobId":"1:BASE_PARITY:12f252fdc7480992"} 19:38:23 [19:38:23.941] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testKeXnbi/2 with map size 1073741824 19:38:23 [19:38:23.942] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.959] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5ca301562ade1571 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5ca301562ade1571"} 19:38:23 [19:38:23.960] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testKeXnbi/1 with map size 1073741824 19:38:23 [19:38:23.963] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:23 [19:38:23.985] INFO: prover-client:proving-broker Proving Broker started 19:38:23 [19:38:23.985] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:071366bd286fdc3b epochNumber=1 {"provingJobId":"1:BASE_PARITY:071366bd286fdc3b"} 19:38:23 [19:38:23.986] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testH2Hp6m/1 with map size 1073741824 19:38:23 [19:38:23.990] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.013] INFO: prover-client:proving-broker Proving Broker started 19:38:24 [19:38:24.013] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b92201774b93ed92 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b92201774b93ed92"} 19:38:24 [19:38:24.014] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlTzOSh/1 with map size 1073741824 19:38:24 [19:38:24.016] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.036] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4d9541b98c33e88d epochNumber=1 {"provingJobId":"1:BASE_PARITY:4d9541b98c33e88d"} 19:38:24 [19:38:24.042] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cdea00394da46032 epochNumber=2 {"provingJobId":"1:BASE_PARITY:cdea00394da46032"} 19:38:24 [19:38:24.043] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlTzOSh/2 with map size 1073741824 19:38:24 [19:38:24.044] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.063] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4b69cda1c8c2d149 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4b69cda1c8c2d149"} 19:38:24 [19:38:24.075] INFO: prover-client:proving-broker Proving Broker started 19:38:24 [19:38:24.076] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:056e6c91b06e0fcf epochNumber=1 {"provingJobId":"1:BASE_PARITY:056e6c91b06e0fcf"} 19:38:24 [19:38:24.077] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbT8SfI/1 with map size 1073741824 19:38:24 [19:38:24.078] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.097] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7e319ef7d568cb95 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7e319ef7d568cb95"} 19:38:24 [19:38:24.103] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:778515eb0c6b9920 epochNumber=2 {"provingJobId":"1:BASE_PARITY:778515eb0c6b9920"} 19:38:24 [19:38:24.104] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testbT8SfI/2 with map size 1073741824 19:38:24 [19:38:24.105] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.122] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:744374de5d00f1a9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:744374de5d00f1a9"} 19:38:24 [19:38:24.133] INFO: prover-client:proving-broker Proving Broker started 19:38:24 [19:38:24.134] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cf9d667675776eaf epochNumber=1 {"provingJobId":"1:BASE_PARITY:cf9d667675776eaf"} 19:38:24 [19:38:24.134] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDELIaa/1 with map size 1073741824 19:38:24 [19:38:24.136] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.152] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:21b8d71405e5a1fa epochNumber=1 {"provingJobId":"1:BASE_PARITY:21b8d71405e5a1fa"} 19:38:24 [19:38:24.157] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4dd73af145937e75 epochNumber=2 {"provingJobId":"1:BASE_PARITY:4dd73af145937e75"} 19:38:24 [19:38:24.158] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testDELIaa/2 with map size 1073741824 19:38:24 [19:38:24.160] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.176] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9f77d9076d771e90 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9f77d9076d771e90"} 19:38:24 [19:38:24.186] INFO: prover-client:proving-broker Proving Broker started 19:38:24 [19:38:24.186] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f6a7f019ce194c9f epochNumber=1 {"provingJobId":"1:BASE_PARITY:f6a7f019ce194c9f"} 19:38:24 [19:38:24.187] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLN2a9S/1 with map size 1073741824 19:38:24 [19:38:24.189] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.206] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f611a5e521c45c6d epochNumber=1 {"provingJobId":"1:BASE_PARITY:f611a5e521c45c6d"} 19:38:24 [19:38:24.212] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f6a7f019ce194c9f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f6a7f019ce194c9f"} 19:38:24 [19:38:24.223] INFO: prover-client:proving-broker Proving Broker started 19:38:24 [19:38:24.224] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:90fd4e6482af6140 epochNumber=1 {"provingJobId":"1:BASE_PARITY:90fd4e6482af6140"} 19:38:24 [19:38:24.225] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test99gDJb/1 with map size 1073741824 19:38:24 [19:38:24.227] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.246] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:929f2d0c244f5570 epochNumber=1 {"provingJobId":"1:BASE_PARITY:929f2d0c244f5570"} 19:38:24 [19:38:24.253] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:90fd4e6482af6140 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:90fd4e6482af6140"} 19:38:24 [19:38:24.262] INFO: prover-client:proving-broker Proving Broker started 19:38:24 [19:38:24.262] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:de1cd99d92841fc6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:de1cd99d92841fc6"} 19:38:24 [19:38:24.263] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3Aaixd/1 with map size 1073741824 19:38:24 [19:38:24.267] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.282] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7e9bc92621a8e4bb epochNumber=1 {"provingJobId":"1:BASE_PARITY:7e9bc92621a8e4bb"} 19:38:24 [19:38:24.287] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:de1cd99d92841fc6 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:de1cd99d92841fc6"} 19:38:24 [19:38:24.291] INFO: prover-client:proving-broker Proving Broker started 19:38:24 [19:38:24.292] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:52e18defe629322b epochNumber=1 {"provingJobId":"1:BASE_PARITY:52e18defe629322b"} 19:38:24 [19:38:24.293] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMU3nIg/1 with map size 1073741824 19:38:24 [19:38:24.295] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.310] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:52e18defe629322b {"provingJobId":"1:BASE_PARITY:52e18defe629322b"} 19:38:24 [19:38:24.311] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:52e18defe629322b type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:52e18defe629322b"} 19:38:24 [19:38:24.318] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:869e4c7fc3fbb030 epochNumber=1 {"provingJobId":"1:BASE_PARITY:869e4c7fc3fbb030"} 19:38:24 [19:38:24.324] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:52e18defe629322b has already been completed {"provingJobId":"1:BASE_PARITY:52e18defe629322b"} 19:38:24 [19:38:24.327] INFO: prover-client:proving-broker Proving Broker started 19:38:24 [19:38:24.328] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e0afec018636ca53 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e0afec018636ca53"} 19:38:24 [19:38:24.329] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testg30EY5/1 with map size 1073741824 19:38:24 [19:38:24.332] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:24 [19:38:24.353] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c83a1945dec63b4e epochNumber=2 {"provingJobId":"1:BASE_PARITY:c83a1945dec63b4e"} 19:38:24 [19:38:24.353] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testg30EY5/2 with map size 1073741824 19:38:24 [19:38:24.355] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:25 [19:38:25.428] INFO: prover-client:proving-broker Proving Broker started 19:38:25 [19:38:25.433] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e0afec018636ca53 settled=false {"provingJobId":"1:BASE_PARITY:e0afec018636ca53","status":"pending"} 19:38:25 [19:38:25.435] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c83a1945dec63b4e settled=false {"provingJobId":"1:BASE_PARITY:c83a1945dec63b4e","status":"pending"} 19:38:25 [19:38:25.436] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:e0afec018636ca53 type=BASE_PARITY startedAt=1750361904376 older agent has taken job {"provingJobId":"1:BASE_PARITY:e0afec018636ca53"} 19:38:25 [19:38:25.437] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e0afec018636ca53 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:e0afec018636ca53"} 19:38:25 [19:38:25.441] INFO: prover-client:proving-broker Proving Broker started 19:38:25 [19:38:25.441] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f36fc19c5de237bc epochNumber=1 {"provingJobId":"1:BASE_PARITY:f36fc19c5de237bc"} 19:38:25 [19:38:25.442] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWIjXB8/1 with map size 1073741824 19:38:25 [19:38:25.444] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:25 [19:38:25.461] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:91f59f8fd44469f4 epochNumber=2 {"provingJobId":"1:BASE_PARITY:91f59f8fd44469f4"} 19:38:25 [19:38:25.462] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWIjXB8/2 with map size 1073741824 19:38:25 [19:38:25.463] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:26 [19:38:26.481] INFO: prover-client:proving-broker Proving Broker started 19:38:26 [19:38:26.483] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f36fc19c5de237bc settled=false {"provingJobId":"1:BASE_PARITY:f36fc19c5de237bc","status":"pending"} 19:38:26 [19:38:26.484] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:91f59f8fd44469f4 settled=false {"provingJobId":"1:BASE_PARITY:91f59f8fd44469f4","status":"pending"} 19:38:26 [19:38:26.485] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f36fc19c5de237bc type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:f36fc19c5de237bc"} 19:38:26 [19:38:26.489] INFO: prover-client:proving-broker Proving Broker started 19:38:26 [19:38:26.489] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d424bd9eb6a6ec28 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d424bd9eb6a6ec28"} 19:38:26 [19:38:26.490] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testn13nP1/1 with map size 1073741824 19:38:26 [19:38:26.492] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:26 [19:38:26.510] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9ecdcb32dae18592 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9ecdcb32dae18592"} 19:38:26 [19:38:26.511] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testn13nP1/2 with map size 1073741824 19:38:26 [19:38:26.512] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:27 [19:38:27.530] INFO: prover-client:proving-broker Proving Broker started 19:38:27 [19:38:27.532] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d424bd9eb6a6ec28 settled=false {"provingJobId":"1:BASE_PARITY:d424bd9eb6a6ec28","status":"pending"} 19:38:27 [19:38:27.534] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9ecdcb32dae18592 settled=false {"provingJobId":"1:BASE_PARITY:9ecdcb32dae18592","status":"pending"} 19:38:27 [19:38:27.535] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d424bd9eb6a6ec28 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d424bd9eb6a6ec28"} 19:38:27 [19:38:27.535] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d424bd9eb6a6ec28 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d424bd9eb6a6ec28"} 19:38:27 [19:38:27.546] INFO: prover-client:proving-broker Proving Broker started 19:38:27 [19:38:27.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0bd735ed6e848f32 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0bd735ed6e848f32"} 19:38:27 [19:38:27.547] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOj0a12/1 with map size 1073741824 19:38:27 [19:38:27.549] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:27 [19:38:27.564] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5090d3ab3805ad18 epochNumber=2 {"provingJobId":"1:BASE_PARITY:5090d3ab3805ad18"} 19:38:27 [19:38:27.565] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOj0a12/2 with map size 1073741824 19:38:27 [19:38:27.567] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:27 [19:38:27.583] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0bd735ed6e848f32 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0bd735ed6e848f32"} 19:38:27 [19:38:27.590] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5090d3ab3805ad18 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:5090d3ab3805ad18"} 19:38:27 [19:38:27.601] INFO: prover-client:proving-broker Proving Broker started 19:38:27 [19:38:27.603] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:deac86e10344905a epochNumber=1 {"provingJobId":"1:BASE_PARITY:deac86e10344905a"} 19:38:27 [19:38:27.603] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrdpZRw/1 with map size 1073741824 19:38:27 [19:38:27.605] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:27 [19:38:27.623] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:428cb1cf8f20d8f4 epochNumber=2 {"provingJobId":"1:BASE_PARITY:428cb1cf8f20d8f4"} 19:38:27 [19:38:27.624] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrdpZRw/2 with map size 1073741824 19:38:27 [19:38:27.627] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:27 [19:38:27.644] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:deac86e10344905a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:deac86e10344905a"} 19:38:27 [19:38:27.644] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:deac86e10344905a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:deac86e10344905a"} 19:38:27 [19:38:27.651] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:428cb1cf8f20d8f4 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:428cb1cf8f20d8f4"} 19:38:27 [19:38:27.659] INFO: prover-client:proving-broker Proving Broker started 19:38:27 [19:38:27.660] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:35fe8468a74dade8 err=test error {"provingJoId":"1:BASE_PARITY:35fe8468a74dade8"} 19:38:27 [19:38:27.662] INFO: prover-client:proving-broker Proving Broker started 19:38:27 [19:38:27.663] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:2178230e64968432 not found {"provingJobId":"1:BASE_PARITY:2178230e64968432"} 19:38:27 [19:38:27.666] INFO: prover-client:proving-broker Proving Broker started 19:38:27 [19:38:27.667] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c0a33700d8378129 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c0a33700d8378129"} 19:38:27 [19:38:27.668] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNNq212/1 with map size 1073741824 19:38:27 [19:38:27.669] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:27 [19:38:27.690] INFO: prover-client:proving-broker Proving Broker started 19:38:27 [19:38:27.690] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c81a1141fbacf2ff epochNumber=1 {"provingJobId":"1:BASE_PARITY:c81a1141fbacf2ff"} 19:38:27 [19:38:27.691] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXLD5Fc/1 with map size 1073741824 19:38:27 [19:38:27.693] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:27 [19:38:27.815] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c81a1141fbacf2ff timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:c81a1141fbacf2ff"} 19:38:27 [19:38:27.820] INFO: prover-client:proving-broker Proving Broker started 19:38:27 [19:38:27.820] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:94e14c3b14298c85 epochNumber=1 {"provingJobId":"1:BASE_PARITY:94e14c3b14298c85"} 19:38:27 [19:38:27.821] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFED9LT/1 with map size 1073741824 19:38:27 [19:38:27.822] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:27 [19:38:27.946] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:94e14c3b14298c85 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:94e14c3b14298c85"} 19:38:27 [19:38:27.952] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f0e8f971efb020ce epochNumber=10 {"provingJobId":"1:BASE_PARITY:f0e8f971efb020ce"} 19:38:27 [19:38:27.953] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testFED9LT/10 with map size 1073741824 19:38:27 [19:38:27.955] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:27 [19:38:27.996] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:27 [19:38:27.997] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 19:38:28 [19:38:28.103] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.104] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3300f18b82f99766 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3300f18b82f99766"} 19:38:28 [19:38:28.104] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5a2tQv/1 with map size 1073741824 19:38:28 [19:38:28.109] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.304] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3300f18b82f99766 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:3300f18b82f99766"} 19:38:28 [19:38:28.336] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.336] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3d874142dd33d408 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3d874142dd33d408"} 19:38:28 [19:38:28.337] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testX4c41p/1 with map size 1073741824 19:38:28 [19:38:28.339] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.357] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3d874142dd33d408 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:3d874142dd33d408"} 19:38:28 [19:38:28.362] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.363] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:41a4ca824bf39f2c epochNumber=1 {"provingJobId":"1:BASE_PARITY:41a4ca824bf39f2c"} 19:38:28 [19:38:28.364] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testv1nusc/1 with map size 1073741824 19:38:28 [19:38:28.366] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.383] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:41a4ca824bf39f2c type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:41a4ca824bf39f2c"} 19:38:28 [19:38:28.384] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:41a4ca824bf39f2c type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:41a4ca824bf39f2c"} 19:38:28 [19:38:28.393] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.394] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ba7bc09ef42da469 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ba7bc09ef42da469"} 19:38:28 [19:38:28.395] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUcnSjq/1 with map size 1073741824 19:38:28 [19:38:28.397] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.413] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:ba7bc09ef42da469 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:ba7bc09ef42da469"} 19:38:28 [19:38:28.423] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.424] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e4aa39f164b54df0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e4aa39f164b54df0"} 19:38:28 [19:38:28.425] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHHp9cv/1 with map size 1073741824 19:38:28 [19:38:28.427] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.443] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e4aa39f164b54df0 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:e4aa39f164b54df0"} 19:38:28 [19:38:28.444] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2e6a286a1fee7072 epochNumber=3 {"provingJobId":"1:BASE_PARITY:2e6a286a1fee7072"} 19:38:28 [19:38:28.445] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testHHp9cv/3 with map size 1073741824 19:38:28 [19:38:28.447] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.475] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:28 [19:38:28.475] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 19:38:28 [19:38:28.488] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e4aa39f164b54df0 err=test error {"provingJoId":"1:BASE_PARITY:e4aa39f164b54df0"} 19:38:28 [19:38:28.491] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testW0Whg7/1 with map size 1073741824 19:38:28 [19:38:28.493] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.509] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testW0Whg7/2 with map size 1073741824 19:38:28 [19:38:28.511] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.526] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.528] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5c357ebf35d149c7 settled=false {"provingJobId":"1:BASE_PARITY:5c357ebf35d149c7","status":"pending"} 19:38:28 [19:38:28.530] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:981e0d3acabe89ab settled=false {"provingJobId":"1:BASE_PARITY:981e0d3acabe89ab","status":"pending"} 19:38:28 [19:38:28.537] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkqqDTS/1 with map size 1073741824 19:38:28 [19:38:28.539] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.555] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkqqDTS/2 with map size 1073741824 19:38:28 [19:38:28.557] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.583] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.585] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a0ac3f62a92a06a6 settled=true {"provingJobId":"1:BASE_PARITY:a0ac3f62a92a06a6","status":"fulfilled"} 19:38:28 [19:38:28.587] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b9314129d8c04e9d settled=true {"provingJobId":"2:BASE_PARITY:b9314129d8c04e9d","status":"fulfilled"} 19:38:28 [19:38:28.593] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTftXpZ/1 with map size 1073741824 19:38:28 [19:38:28.595] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.616] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTftXpZ/2 with map size 1073741824 19:38:28 [19:38:28.619] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.634] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.636] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e5ce1abee241ca29 settled=true {"provingJobId":"1:BASE_PARITY:e5ce1abee241ca29","status":"fulfilled"} 19:38:28 [19:38:28.637] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9c7f6e2bca8d7821 settled=false {"provingJobId":"1:BASE_PARITY:9c7f6e2bca8d7821","status":"pending"} 19:38:28 [19:38:28.641] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.642] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7ea7ef0226d4135a epochNumber=1 {"provingJobId":"1:BASE_PARITY:7ea7ef0226d4135a"} 19:38:28 [19:38:28.642] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testW5FzDU/1 with map size 1073741824 19:38:28 [19:38:28.644] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.663] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.663] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4f299a7ce20c3a1d epochNumber=1 {"provingJobId":"1:BASE_PARITY:4f299a7ce20c3a1d"} 19:38:28 [19:38:28.664] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:4f299a7ce20c3a1d: Error: db error: Error: db error 19:38:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:4f299a7ce20c3a1d"} 19:38:28 [19:38:28.668] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.669] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e41d38238429058a epochNumber=1 {"provingJobId":"1:BASE_PARITY:e41d38238429058a"} 19:38:28 [19:38:28.670] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9e2JdU/1 with map size 1073741824 19:38:28 [19:38:28.672] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.687] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e41d38238429058a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e41d38238429058a"} 19:38:28 [19:38:28.688] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e41d38238429058a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e41d38238429058a"} 19:38:28 [19:38:28.697] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.698] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c1160969c3d30f74 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c1160969c3d30f74"} 19:38:28 [19:38:28.699] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testES02XU/1 with map size 1073741824 19:38:28 [19:38:28.701] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.717] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c1160969c3d30f74 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c1160969c3d30f74"} 19:38:28 [19:38:28.717] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c1160969c3d30f74 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:c1160969c3d30f74"} 19:38:28 [19:38:28.718] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:c1160969c3d30f74: Error: db error 19:38:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:c1160969c3d30f74"} 19:38:28 [19:38:28.723] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.723] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:50cbe0135e008f14 epochNumber=1 {"provingJobId":"1:BASE_PARITY:50cbe0135e008f14"} 19:38:28 [19:38:28.724] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOYG21i/1 with map size 1073741824 19:38:28 [19:38:28.727] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.742] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:50cbe0135e008f14 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:50cbe0135e008f14"} 19:38:28 [19:38:28.743] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:50cbe0135e008f14 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:50cbe0135e008f14"} 19:38:28 [19:38:28.755] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.755] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3238aebb85ad9d2c epochNumber=1 {"provingJobId":"1:BASE_PARITY:3238aebb85ad9d2c"} 19:38:28 [19:38:28.756] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1drToH/1 with map size 1073741824 19:38:28 [19:38:28.758] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.774] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3238aebb85ad9d2c type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3238aebb85ad9d2c"} 19:38:28 [19:38:28.774] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3238aebb85ad9d2c type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:3238aebb85ad9d2c"} 19:38:28 [19:38:28.775] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:3238aebb85ad9d2c jobErr=test error: Error: db error 19:38:28 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:3238aebb85ad9d2c"} 19:38:28 [19:38:28.780] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.781] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7fd0bcfc4f7e6d1d not found {"provingJobId":"1:BASE_PARITY:7fd0bcfc4f7e6d1d"} 19:38:28 [19:38:28.785] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.786] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:04f915d7cf110b42 err=test error {"provingJoId":"1:BASE_PARITY:04f915d7cf110b42"} 19:38:28 [19:38:28.790] INFO: prover-client:proving-broker Proving Broker started 19:38:28 [19:38:28.800] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:09ba6141a440a633 epochNumber=1 {"provingJobId":"1:BASE_PARITY:09ba6141a440a633"} 19:38:28 [19:38:28.801] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlDm8Gv/1 with map size 1073741824 19:38:28 [19:38:28.803] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.819] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:09ba6141a440a633 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:09ba6141a440a633"} 19:38:28 [19:38:28.819] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:09ba6141a440a633 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:09ba6141a440a633"} 19:38:28 [19:38:28.835] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:0c5e1514c8042e8f epochNumber=2 {"provingJobId":"2:BASE_PARITY:0c5e1514c8042e8f"} 19:38:28 [19:38:28.836] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlDm8Gv/2 with map size 1073741824 19:38:28 [19:38:28.838] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.853] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:0c5e1514c8042e8f type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:0c5e1514c8042e8f"} 19:38:28 [19:38:28.854] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:0c5e1514c8042e8f type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:0c5e1514c8042e8f"} 19:38:28 [19:38:28.920] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:8e5e45e321eae571 epochNumber=3 {"provingJobId":"3:BASE_PARITY:8e5e45e321eae571"} 19:38:28 [19:38:28.921] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testlDm8Gv/3 with map size 1073741824 19:38:28 [19:38:28.925] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:28 [19:38:28.968] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:28 [19:38:28.969] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 19:38:29 [19:38:29.009] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:55011877f78b59aa epochNumber=4 {"provingJobId":"4:BASE_PARITY:55011877f78b59aa"} 19:38:29 [19:38:29.010] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testlDm8Gv/4 with map size 1073741824 19:38:29 [19:38:29.013] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:29 [19:38:29.046] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:29 [19:38:29.047] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 19:38:29 [19:38:29.095] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:f80a565f06a9cf8d epochNumber=5 {"provingJobId":"5:BASE_PARITY:f80a565f06a9cf8d"} 19:38:29 [19:38:29.096] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testlDm8Gv/5 with map size 1073741824 19:38:29 [19:38:29.099] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 19:38:29 [19:38:29.124] INFO: prover-client:proving-broker Cleaned up jobs=1 19:38:29 [19:38:29.125] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 19:38:29 PASS src/proving_broker/proving_broker.test.ts (12.202 s) 19:38:29 ProvingBroker 19:38:29 Producer API 19:38:29 ✓ refuses stale jobs (49 ms) 19:38:29 ✓ enqueues jobs (5 ms) 19:38:29 ✓ ignores duplicate jobs (4 ms) 19:38:29 ✓ reports correct status when enqueuing repeat jobs (4 ms) 19:38:29 ✓ throws an error in case of duplicate job IDs (33 ms) 19:38:29 ✓ returns not-found status for non-existing jobs (2 ms) 19:38:29 ✓ cancels jobs in queue (3 ms) 19:38:29 ✓ cancels jobs in-progress (3 ms) 19:38:29 ✓ returns job result if successful (3 ms) 19:38:29 ✓ returns job error if failed (2 ms) 19:38:29 ✓ correctly returns job status for concurrent writes (3 ms) 19:38:29 ✓ refuses stale jobs (35 ms) 19:38:29 ✓ enqueues jobs (32 ms) 19:38:29 ✓ ignores duplicate jobs (23 ms) 19:38:29 ✓ reports correct status when enqueuing repeat jobs (29 ms) 19:38:29 ✓ throws an error in case of duplicate job IDs (22 ms) 19:38:29 ✓ returns not-found status for non-existing jobs (2 ms) 19:38:29 ✓ cancels jobs in queue (31 ms) 19:38:29 ✓ cancels jobs in-progress (29 ms) 19:38:29 ✓ returns job result if successful (26 ms) 19:38:29 ✓ returns job error if failed (25 ms) 19:38:29 ✓ correctly returns job status for concurrent writes (35 ms) 19:38:29 Consumer API 19:38:29 ✓ returns undefined if no jobs are available (2 ms) 19:38:29 ✓ returns jobs in priority order (2 ms) 19:38:29 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 19:38:29 ✓ returns a job if it is in the allowList (3 ms) 19:38:29 ✓ returns the most important job if it is in the allowList (3 ms) 19:38:29 ✓ returns any job if filter is empty (2 ms) 19:38:29 ✓ returns a new job when reporting job success (3 ms) 19:38:29 ✓ returns a new job when reporting permanent error (2 ms) 19:38:29 ✓ returns a new job when reporting retry-able error (2 ms) 19:38:29 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 19:38:29 ✓ returns a new job if job is already in progress elsewhere (1057 ms) 19:38:29 ✓ avoids sending the same job to a new agent after a restart (1004 ms) 19:38:29 ✓ avoids sending a completed job to a new agent after a restart (1005 ms) 19:38:29 ✓ tracks job result if in progress (3 ms) 19:38:29 ✓ tracks job result even if job is in queue (3 ms) 19:38:29 ✓ ignores reported job error if unknown job (1 ms) 19:38:29 ✓ ignores job result if unknown job (2 ms) 19:38:29 ✓ returns undefined if no jobs are available (2 ms) 19:38:29 ✓ returns jobs in priority order (44 ms) 19:38:29 ✓ returns undefined if no jobs are available for the given allowList (26 ms) 19:38:29 ✓ returns a job if it is in the allowList (64 ms) 19:38:29 ✓ returns the most important job if it is in the allowList (58 ms) 19:38:29 ✓ returns any job if filter is empty (52 ms) 19:38:29 ✓ returns a new job when reporting job success (36 ms) 19:38:29 ✓ returns a new job when reporting permanent error (39 ms) 19:38:29 ✓ returns a new job when reporting retry-able error (29 ms) 19:38:29 ✓ returns a new job when reporting progress if current one is cancelled (36 ms) 19:38:29 ✓ returns a new job if job is already in progress elsewhere (1112 ms) 19:38:29 ✓ avoids sending the same job to a new agent after a restart (1048 ms) 19:38:29 ✓ avoids sending a completed job to a new agent after a restart (1056 ms) 19:38:29 ✓ tracks job result if in progress (55 ms) 19:38:29 ✓ tracks job result even if job is in queue (58 ms) 19:38:29 ✓ ignores reported job error if unknown job (2 ms) 19:38:29 ✓ ignores job result if unknown job (2 ms) 19:38:29 Timeouts 19:38:29 ✓ tracks in progress jobs (2 ms) 19:38:29 ✓ re-enqueues jobs that time out (110 ms) 19:38:29 ✓ cancel stale jobs that time out (229 ms) 19:38:29 ✓ keeps the jobs in progress while it is alive (202 ms) 19:38:29 ✓ tracks in progress jobs (24 ms) 19:38:29 ✓ re-enqueues jobs that time out (129 ms) 19:38:29 ✓ cancel stale jobs that time out (282 ms) 19:38:29 ✓ keeps the jobs in progress while it is alive (234 ms) 19:38:29 Retries 19:38:29 ✓ retries jobs (3 ms) 19:38:29 ✓ retries up to a maximum number of times (3 ms) 19:38:29 ✓ passing retry=false does not retry (3 ms) 19:38:29 ✓ does not retry if job is stale (29 ms) 19:38:29 ✓ retries jobs (26 ms) 19:38:29 ✓ retries up to a maximum number of times (31 ms) 19:38:29 ✓ passing retry=false does not retry (29 ms) 19:38:29 ✓ does not retry if job is stale (69 ms) 19:38:29 Database management 19:38:29 ✓ re-enqueues proof requests on start (3 ms) 19:38:29 ✓ restores proof results on start (2 ms) 19:38:29 ✓ only re-enqueues unfinished jobs (2 ms) 19:38:29 ✓ saves job when enqueued (2 ms) 19:38:29 ✓ does not retain job if database fails to save (2 ms) 19:38:29 ✓ saves job result (1 ms) 19:38:29 ✓ saves result even if database fails to save (2 ms) 19:38:29 ✓ saves job error (2 ms) 19:38:29 ✓ saves job error even if database fails to save (2 ms) 19:38:29 ✓ does not save job result if job is unknown (1 ms) 19:38:29 ✓ does not save job error if job is unknown (1 ms) 19:38:29 ✓ cleans up old jobs periodically (259 ms) 19:38:29 ✓ re-enqueues proof requests on start (44 ms) 19:38:29 ✓ restores proof results on start (55 ms) 19:38:29 ✓ only re-enqueues unfinished jobs (49 ms) 19:38:29 ✓ saves job when enqueued (20 ms) 19:38:29 ✓ does not retain job if database fails to save (4 ms) 19:38:29 ✓ saves job result (29 ms) 19:38:29 ✓ saves result even if database fails to save (25 ms) 19:38:29 ✓ saves job error (31 ms) 19:38:29 ✓ saves job error even if database fails to save (25 ms) 19:38:29 ✓ does not save job result if job is unknown (5 ms) 19:38:29 ✓ does not save job error if job is unknown (4 ms) 19:38:29 ✓ cleans up old jobs periodically (383 ms) 19:38:29 19:38:29 Test Suites: 1 passed, 1 total 19:38:29 Tests: 96 passed, 96 total 19:38:29 Snapshots: 0 total 19:38:29 Time: 12.297 s 19:38:29 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 19:38:29 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?