ACI ·

Parent Log: http://ci.aztec-labs.com/e352c3f7d5a3a836 Command: 17e0d785507e6983 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/5867520ed258adecfee692b3694d693b2c7964ca Env: REF_NAME=gh-readonly-queue/next/pr-14770-9b801754ba98479ee645488c29d2fda1eecbfd85 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Mon Jun 16 21:51:33 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14770_amd64_x1-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 21:51:38 [21:51:38.115] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.123] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d06f53a1f5790a7a epochNumber=42 {"provingJobId":"1:BASE_PARITY:d06f53a1f5790a7a"} 21:51:38 [21:51:38.127] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:94ecb6b8180fe917 epochNumber=1 {"provingJobId":"1:BASE_PARITY:94ecb6b8180fe917"} 21:51:38 [21:51:38.165] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.166] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4eacf9f3a6ec0b1e epochNumber=1 {"provingJobId":"1:BASE_PARITY:4eacf9f3a6ec0b1e"} 21:51:38 [21:51:38.168] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3cc1d36756471140 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3cc1d36756471140"} 21:51:38 [21:51:38.171] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.172] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:693f1b66f053d52f epochNumber=1 {"provingJobId":"1:BASE_PARITY:693f1b66f053d52f"} 21:51:38 [21:51:38.174] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:693f1b66f053d52f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:693f1b66f053d52f"} 21:51:38 [21:51:38.177] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.178] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:48d220e8d7625ed3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:48d220e8d7625ed3"} 21:51:38 [21:51:38.180] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:48d220e8d7625ed3 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:48d220e8d7625ed3"} 21:51:38 [21:51:38.180] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:48d220e8d7625ed3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:48d220e8d7625ed3"} 21:51:38 [21:51:38.181] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:48d220e8d7625ed3 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:48d220e8d7625ed3"} 21:51:38 [21:51:38.183] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.184] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0be0699849529535 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0be0699849529535"} 21:51:38 [21:51:38.230] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.233] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.234] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4968b90103082389 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4968b90103082389"} 21:51:38 [21:51:38.235] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:4968b90103082389 {"provingJobId":"1:BASE_PARITY:4968b90103082389"} 21:51:38 [21:51:38.235] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4968b90103082389 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4968b90103082389"} 21:51:38 [21:51:38.236] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4968b90103082389 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:4968b90103082389"} 21:51:38 [21:51:38.238] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.238] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:84d03dc75079ad60 epochNumber=1 {"provingJobId":"1:BASE_PARITY:84d03dc75079ad60"} 21:51:38 [21:51:38.239] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:84d03dc75079ad60 {"provingJobId":"1:BASE_PARITY:84d03dc75079ad60"} 21:51:38 [21:51:38.240] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:84d03dc75079ad60 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:84d03dc75079ad60"} 21:51:38 [21:51:38.242] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.243] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bce3601c98075efd epochNumber=1 {"provingJobId":"1:BASE_PARITY:bce3601c98075efd"} 21:51:38 [21:51:38.243] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bce3601c98075efd type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:bce3601c98075efd"} 21:51:38 [21:51:38.244] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bce3601c98075efd type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:bce3601c98075efd"} 21:51:38 [21:51:38.246] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.246] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:43db4b6e0b1fd01a epochNumber=1 {"provingJobId":"1:BASE_PARITY:43db4b6e0b1fd01a"} 21:51:38 [21:51:38.247] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:43db4b6e0b1fd01a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:43db4b6e0b1fd01a"} 21:51:38 [21:51:38.247] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:43db4b6e0b1fd01a type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:43db4b6e0b1fd01a"} 21:51:38 [21:51:38.248] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.249] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0c36854867e51dc0 epochNumber=0 {"provingJobId":"1:BASE_PARITY:0c36854867e51dc0"} 21:51:38 [21:51:38.250] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0c36854867e51dc0 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:0c36854867e51dc0"} 21:51:38 [21:51:38.250] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:af0a9c33e44525d4 epochNumber=0 {"provingJobId":"1:BASE_PARITY:af0a9c33e44525d4"} 21:51:38 [21:51:38.250] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:0c36854867e51dc0 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:0c36854867e51dc0"} 21:51:38 [21:51:38.251] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7afc713da692a95e epochNumber=0 {"provingJobId":"1:BASE_PARITY:7afc713da692a95e"} 21:51:38 [21:51:38.252] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.254] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.254] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dfffeea6b08cbbf7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:dfffeea6b08cbbf7"} 21:51:38 [21:51:38.254] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cbbca9e456f4f54c epochNumber=1 {"provingJobId":"1:BASE_PARITY:cbbca9e456f4f54c"} 21:51:38 [21:51:38.256] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.256] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aa9ac8ce6859f118 epochNumber=1 {"provingJobId":"1:BASE_PARITY:aa9ac8ce6859f118"} 21:51:38 [21:51:38.258] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.258] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ab94f30096075e8f epochNumber=1 {"provingJobId":"1:BASE_PARITY:ab94f30096075e8f"} 21:51:38 [21:51:38.259] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b802ab0469951eb0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b802ab0469951eb0"} 21:51:38 [21:51:38.259] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dd0baad016e6fa5c epochNumber=2 {"provingJobId":"1:BASE_PARITY:dd0baad016e6fa5c"} 21:51:38 [21:51:38.259] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cce93027be51f03a epochNumber=1 {"provingJobId":"1:BASE_PARITY:cce93027be51f03a"} 21:51:38 [21:51:38.261] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.261] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a29bdf4e31651643 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a29bdf4e31651643"} 21:51:38 [21:51:38.262] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a458823c35144495 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a458823c35144495"} 21:51:38 [21:51:38.262] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e2917f28c984a855 epochNumber=2 {"provingJobId":"1:BASE_PARITY:e2917f28c984a855"} 21:51:38 [21:51:38.262] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:59a510e77b4d002e epochNumber=1 {"provingJobId":"1:BASE_PARITY:59a510e77b4d002e"} 21:51:38 [21:51:38.264] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.264] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:58debe6a29a1e7de epochNumber=1 {"provingJobId":"1:BASE_PARITY:58debe6a29a1e7de"} 21:51:38 [21:51:38.265] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:73c564c7d7dc2421 epochNumber=1 {"provingJobId":"1:BASE_PARITY:73c564c7d7dc2421"} 21:51:38 [21:51:38.265] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a0897ac4a1777513 epochNumber=2 {"provingJobId":"1:BASE_PARITY:a0897ac4a1777513"} 21:51:38 [21:51:38.265] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:07855ee23de3751d epochNumber=1 {"provingJobId":"1:BASE_PARITY:07855ee23de3751d"} 21:51:38 [21:51:38.268] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.269] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cfdfdc5a0b030587 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cfdfdc5a0b030587"} 21:51:38 [21:51:38.270] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dcb4e0edc919f5b1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dcb4e0edc919f5b1"} 21:51:38 [21:51:38.271] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cfdfdc5a0b030587 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cfdfdc5a0b030587"} 21:51:38 [21:51:38.273] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.274] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f0ba9523ead76e99 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f0ba9523ead76e99"} 21:51:38 [21:51:38.274] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f7b507133f845351 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f7b507133f845351"} 21:51:38 [21:51:38.275] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f0ba9523ead76e99 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:f0ba9523ead76e99"} 21:51:38 [21:51:38.276] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.277] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9fd357100047f59e epochNumber=1 {"provingJobId":"1:BASE_PARITY:9fd357100047f59e"} 21:51:38 [21:51:38.277] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f0a10a61932d2e82 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f0a10a61932d2e82"} 21:51:38 [21:51:38.277] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9fd357100047f59e type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:9fd357100047f59e"} 21:51:38 [21:51:38.279] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.279] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b7a60755650d3aed epochNumber=1 {"provingJobId":"1:BASE_PARITY:b7a60755650d3aed"} 21:51:38 [21:51:38.280] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b7a60755650d3aed {"provingJobId":"1:BASE_PARITY:b7a60755650d3aed"} 21:51:38 [21:51:38.280] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b7a60755650d3aed type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:b7a60755650d3aed"} 21:51:38 [21:51:38.280] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b77f89af844b2d77 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b77f89af844b2d77"} 21:51:38 [21:51:38.281] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b7a60755650d3aed has already been completed {"provingJobId":"1:BASE_PARITY:b7a60755650d3aed"} 21:51:38 [21:51:38.282] INFO: prover-client:proving-broker Proving Broker started 21:51:38 [21:51:38.283] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b98139409b1e8de5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b98139409b1e8de5"} 21:51:38 [21:51:38.284] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:592eacb93952c51e epochNumber=2 {"provingJobId":"1:BASE_PARITY:592eacb93952c51e"} 21:51:39 [21:51:39.337] INFO: prover-client:proving-broker Proving Broker started 21:51:39 [21:51:39.338] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b98139409b1e8de5 settled=false {"provingJobId":"1:BASE_PARITY:b98139409b1e8de5","status":"pending"} 21:51:39 [21:51:39.338] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:592eacb93952c51e settled=false {"provingJobId":"1:BASE_PARITY:592eacb93952c51e","status":"pending"} 21:51:39 [21:51:39.339] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:b98139409b1e8de5 type=BASE_PARITY startedAt=1750110698284 older agent has taken job {"provingJobId":"1:BASE_PARITY:b98139409b1e8de5"} 21:51:39 [21:51:39.340] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b98139409b1e8de5 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:b98139409b1e8de5"} 21:51:39 [21:51:39.341] INFO: prover-client:proving-broker Proving Broker started 21:51:39 [21:51:39.342] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d21f457c348f3d6a epochNumber=1 {"provingJobId":"1:BASE_PARITY:d21f457c348f3d6a"} 21:51:39 [21:51:39.342] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:41aa907187d05650 epochNumber=2 {"provingJobId":"1:BASE_PARITY:41aa907187d05650"} 21:51:40 [21:51:40.343] INFO: prover-client:proving-broker Proving Broker started 21:51:40 [21:51:40.344] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d21f457c348f3d6a settled=false {"provingJobId":"1:BASE_PARITY:d21f457c348f3d6a","status":"pending"} 21:51:40 [21:51:40.344] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:41aa907187d05650 settled=false {"provingJobId":"1:BASE_PARITY:41aa907187d05650","status":"pending"} 21:51:40 [21:51:40.345] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d21f457c348f3d6a type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:d21f457c348f3d6a"} 21:51:40 [21:51:40.346] INFO: prover-client:proving-broker Proving Broker started 21:51:40 [21:51:40.347] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5e27f2063c60b687 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5e27f2063c60b687"} 21:51:40 [21:51:40.348] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8546d8d81fb82488 epochNumber=2 {"provingJobId":"1:BASE_PARITY:8546d8d81fb82488"} 21:51:41 [21:51:41.348] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.349] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5e27f2063c60b687 settled=false {"provingJobId":"1:BASE_PARITY:5e27f2063c60b687","status":"pending"} 21:51:41 [21:51:41.349] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8546d8d81fb82488 settled=false {"provingJobId":"1:BASE_PARITY:8546d8d81fb82488","status":"pending"} 21:51:41 [21:51:41.350] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5e27f2063c60b687 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5e27f2063c60b687"} 21:51:41 [21:51:41.350] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5e27f2063c60b687 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5e27f2063c60b687"} 21:51:41 [21:51:41.353] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.354] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7134ecbad4fe3303 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7134ecbad4fe3303"} 21:51:41 [21:51:41.354] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d0be1fbb7d600b9f epochNumber=2 {"provingJobId":"1:BASE_PARITY:d0be1fbb7d600b9f"} 21:51:41 [21:51:41.354] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7134ecbad4fe3303 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:7134ecbad4fe3303"} 21:51:41 [21:51:41.355] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d0be1fbb7d600b9f type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:d0be1fbb7d600b9f"} 21:51:41 [21:51:41.356] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.356] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0fa67097ce368991 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0fa67097ce368991"} 21:51:41 [21:51:41.357] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:19d4082c0249c1a7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:19d4082c0249c1a7"} 21:51:41 [21:51:41.358] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0fa67097ce368991 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0fa67097ce368991"} 21:51:41 [21:51:41.358] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0fa67097ce368991 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0fa67097ce368991"} 21:51:41 [21:51:41.358] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:19d4082c0249c1a7 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:19d4082c0249c1a7"} 21:51:41 [21:51:41.359] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.360] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:1a30d87bde7ea1e5 err=test error {"provingJoId":"1:BASE_PARITY:1a30d87bde7ea1e5"} 21:51:41 [21:51:41.361] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.361] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:59c2238477ce1f40 not found {"provingJobId":"1:BASE_PARITY:59c2238477ce1f40"} 21:51:41 [21:51:41.363] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.364] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1e4730b9808b56ac epochNumber=1 {"provingJobId":"1:BASE_PARITY:1e4730b9808b56ac"} 21:51:41 [21:51:41.365] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.366] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:51b7e802c9e2da4e epochNumber=1 {"provingJobId":"1:BASE_PARITY:51b7e802c9e2da4e"} 21:51:41 [21:51:41.492] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:51b7e802c9e2da4e timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:51b7e802c9e2da4e"} 21:51:41 [21:51:41.496] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.496] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f74f33890089faf2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f74f33890089faf2"} 21:51:41 [21:51:41.598] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f74f33890089faf2 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:f74f33890089faf2"} 21:51:41 [21:51:41.605] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2943dc679709d401 epochNumber=10 {"provingJobId":"1:BASE_PARITY:2943dc679709d401"} 21:51:41 [21:51:41.624] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:41 [21:51:41.732] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.733] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4a15ba1d5c3d0d67 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4a15ba1d5c3d0d67"} 21:51:41 [21:51:41.888] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4a15ba1d5c3d0d67 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:4a15ba1d5c3d0d67"} 21:51:41 [21:51:41.938] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.938] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f4298030bf81dd75 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f4298030bf81dd75"} 21:51:41 [21:51:41.939] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f4298030bf81dd75 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:f4298030bf81dd75"} 21:51:41 [21:51:41.941] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.941] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d0c56dd20dc5680f epochNumber=1 {"provingJobId":"1:BASE_PARITY:d0c56dd20dc5680f"} 21:51:41 [21:51:41.942] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d0c56dd20dc5680f type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:d0c56dd20dc5680f"} 21:51:41 [21:51:41.943] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d0c56dd20dc5680f type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:d0c56dd20dc5680f"} 21:51:41 [21:51:41.944] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.945] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f576aae02dbec30b epochNumber=1 {"provingJobId":"1:BASE_PARITY:f576aae02dbec30b"} 21:51:41 [21:51:41.945] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f576aae02dbec30b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:f576aae02dbec30b"} 21:51:41 [21:51:41.946] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.947] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d86900f2b17ad36a epochNumber=1 {"provingJobId":"1:BASE_PARITY:d86900f2b17ad36a"} 21:51:41 [21:51:41.947] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d86900f2b17ad36a type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:d86900f2b17ad36a"} 21:51:41 [21:51:41.948] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0179abe01a09a6cf epochNumber=3 {"provingJobId":"1:BASE_PARITY:0179abe01a09a6cf"} 21:51:41 [21:51:41.973] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:41 [21:51:41.974] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d86900f2b17ad36a err=test error {"provingJoId":"1:BASE_PARITY:d86900f2b17ad36a"} 21:51:41 [21:51:41.976] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.976] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fd9f2ef26671f2f5 settled=false {"provingJobId":"1:BASE_PARITY:fd9f2ef26671f2f5","status":"pending"} 21:51:41 [21:51:41.976] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4eb392f5b9cbf942 settled=false {"provingJobId":"1:BASE_PARITY:4eb392f5b9cbf942","status":"pending"} 21:51:41 [21:51:41.979] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.979] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fdf0ac2dc933db91 settled=true {"provingJobId":"1:BASE_PARITY:fdf0ac2dc933db91","status":"fulfilled"} 21:51:41 [21:51:41.979] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b8da568a1ff25b8a settled=true {"provingJobId":"2:BASE_PARITY:b8da568a1ff25b8a","status":"fulfilled"} 21:51:41 [21:51:41.981] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.981] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c3aa86faf41e28d4 settled=true {"provingJobId":"1:BASE_PARITY:c3aa86faf41e28d4","status":"fulfilled"} 21:51:41 [21:51:41.981] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2a9928511e4e14c0 settled=false {"provingJobId":"1:BASE_PARITY:2a9928511e4e14c0","status":"pending"} 21:51:41 [21:51:41.983] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.984] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:328d3c975a8780c9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:328d3c975a8780c9"} 21:51:41 [21:51:41.986] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.986] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f82785804e68c831 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f82785804e68c831"} 21:51:41 [21:51:41.987] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:f82785804e68c831: Error: db error: Error: db error 21:51:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:f82785804e68c831"} 21:51:41 [21:51:41.988] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.989] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:251c53ad0c5bcf28 epochNumber=1 {"provingJobId":"1:BASE_PARITY:251c53ad0c5bcf28"} 21:51:41 [21:51:41.989] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:251c53ad0c5bcf28 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:251c53ad0c5bcf28"} 21:51:41 [21:51:41.989] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:251c53ad0c5bcf28 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:251c53ad0c5bcf28"} 21:51:41 [21:51:41.990] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.991] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5f8716cb872955fd epochNumber=1 {"provingJobId":"1:BASE_PARITY:5f8716cb872955fd"} 21:51:41 [21:51:41.991] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5f8716cb872955fd type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5f8716cb872955fd"} 21:51:41 [21:51:41.991] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5f8716cb872955fd type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5f8716cb872955fd"} 21:51:41 [21:51:41.992] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:5f8716cb872955fd: Error: db error 21:51:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:5f8716cb872955fd"} 21:51:41 [21:51:41.993] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.993] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bda3aef4bb3218bd epochNumber=1 {"provingJobId":"1:BASE_PARITY:bda3aef4bb3218bd"} 21:51:41 [21:51:41.994] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bda3aef4bb3218bd type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:bda3aef4bb3218bd"} 21:51:41 [21:51:41.994] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bda3aef4bb3218bd type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:bda3aef4bb3218bd"} 21:51:41 [21:51:41.995] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.995] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b8f5c240028972a5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b8f5c240028972a5"} 21:51:41 [21:51:41.995] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b8f5c240028972a5 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b8f5c240028972a5"} 21:51:41 [21:51:41.996] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b8f5c240028972a5 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b8f5c240028972a5"} 21:51:41 [21:51:41.996] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:b8f5c240028972a5 jobErr=test error: Error: db error 21:51:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:b8f5c240028972a5"} 21:51:41 [21:51:41.997] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.997] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8fac450513f5864f not found {"provingJobId":"1:BASE_PARITY:8fac450513f5864f"} 21:51:41 [21:51:41.998] INFO: prover-client:proving-broker Proving Broker started 21:51:41 [21:51:41.999] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6a0ce3082754c0f1 err=test error {"provingJoId":"1:BASE_PARITY:6a0ce3082754c0f1"} 21:51:41 [21:51:42.000] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.010] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1080ac737e112ace epochNumber=1 {"provingJobId":"1:BASE_PARITY:1080ac737e112ace"} 21:51:42 [21:51:42.010] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1080ac737e112ace type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:1080ac737e112ace"} 21:51:42 [21:51:42.010] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1080ac737e112ace type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:1080ac737e112ace"} 21:51:42 [21:51:42.022] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:711a319f8f29987d epochNumber=2 {"provingJobId":"2:BASE_PARITY:711a319f8f29987d"} 21:51:42 [21:51:42.022] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:711a319f8f29987d type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:711a319f8f29987d"} 21:51:42 [21:51:42.023] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:711a319f8f29987d type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:711a319f8f29987d"} 21:51:42 [21:51:42.085] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:c3264a06649a89e7 epochNumber=3 {"provingJobId":"3:BASE_PARITY:c3264a06649a89e7"} 21:51:42 [21:51:42.103] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:42 [21:51:42.146] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:352461bbb36ced5d epochNumber=4 {"provingJobId":"4:BASE_PARITY:352461bbb36ced5d"} 21:51:42 [21:51:42.153] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:42 [21:51:42.208] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:d0291e6810af7530 epochNumber=5 {"provingJobId":"5:BASE_PARITY:d0291e6810af7530"} 21:51:42 [21:51:42.230] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:42 [21:51:42.262] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.263] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d1a4691b2dd6db3e epochNumber=42 {"provingJobId":"1:BASE_PARITY:d1a4691b2dd6db3e"} 21:51:42 [21:51:42.265] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testff710l/42 with map size 1073741824 21:51:42 [21:51:42.271] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.302] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:92660a66240f517f epochNumber=1 {"provingJobId":"1:BASE_PARITY:92660a66240f517f"} 21:51:42 [21:51:42.306] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.307] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0bd9930937b6fe36 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0bd9930937b6fe36"} 21:51:42 [21:51:42.308] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testglqTv7/1 with map size 1073741824 21:51:42 [21:51:42.315] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.334] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2bc511195700f942 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2bc511195700f942"} 21:51:42 [21:51:42.344] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.345] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e9167f1b533c0a31 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e9167f1b533c0a31"} 21:51:42 [21:51:42.346] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdIXsYi/1 with map size 1073741824 21:51:42 [21:51:42.348] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.366] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:e9167f1b533c0a31 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:e9167f1b533c0a31"} 21:51:42 [21:51:42.375] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.376] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eba35a0bfd03a60f epochNumber=1 {"provingJobId":"1:BASE_PARITY:eba35a0bfd03a60f"} 21:51:42 [21:51:42.377] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8V7AcE/1 with map size 1073741824 21:51:42 [21:51:42.379] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.400] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:eba35a0bfd03a60f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:eba35a0bfd03a60f"} 21:51:42 [21:51:42.400] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:eba35a0bfd03a60f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:eba35a0bfd03a60f"} 21:51:42 [21:51:42.407] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:eba35a0bfd03a60f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:eba35a0bfd03a60f"} 21:51:42 [21:51:42.411] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.412] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8d611114aa2be48a epochNumber=1 {"provingJobId":"1:BASE_PARITY:8d611114aa2be48a"} 21:51:42 [21:51:42.412] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsEwVcR/1 with map size 1073741824 21:51:42 [21:51:42.415] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.439] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.442] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.443] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e39b05036d6ee6cc epochNumber=1 {"provingJobId":"1:BASE_PARITY:e39b05036d6ee6cc"} 21:51:42 [21:51:42.443] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhdff3D/1 with map size 1073741824 21:51:42 [21:51:42.446] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.464] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e39b05036d6ee6cc {"provingJobId":"1:BASE_PARITY:e39b05036d6ee6cc"} 21:51:42 [21:51:42.464] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e39b05036d6ee6cc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e39b05036d6ee6cc"} 21:51:42 [21:51:42.465] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e39b05036d6ee6cc type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:e39b05036d6ee6cc"} 21:51:42 [21:51:42.475] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.476] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:da3db7c35389895b epochNumber=1 {"provingJobId":"1:BASE_PARITY:da3db7c35389895b"} 21:51:42 [21:51:42.476] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrFtYvv/1 with map size 1073741824 21:51:42 [21:51:42.479] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.500] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:da3db7c35389895b {"provingJobId":"1:BASE_PARITY:da3db7c35389895b"} 21:51:42 [21:51:42.500] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:da3db7c35389895b type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:da3db7c35389895b"} 21:51:42 [21:51:42.511] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.512] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c80aa635f4011038 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c80aa635f4011038"} 21:51:42 [21:51:42.513] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnmm1z5/1 with map size 1073741824 21:51:42 [21:51:42.515] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.534] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c80aa635f4011038 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c80aa635f4011038"} 21:51:42 [21:51:42.535] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c80aa635f4011038 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:c80aa635f4011038"} 21:51:42 [21:51:42.543] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.544] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a2dfc62b6b6d6c7a epochNumber=1 {"provingJobId":"1:BASE_PARITY:a2dfc62b6b6d6c7a"} 21:51:42 [21:51:42.544] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMqVgVT/1 with map size 1073741824 21:51:42 [21:51:42.546] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.562] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a2dfc62b6b6d6c7a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a2dfc62b6b6d6c7a"} 21:51:42 [21:51:42.563] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a2dfc62b6b6d6c7a type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:a2dfc62b6b6d6c7a"} 21:51:42 [21:51:42.572] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.572] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03705bacd2e68ba9 epochNumber=0 {"provingJobId":"1:BASE_PARITY:03705bacd2e68ba9"} 21:51:42 [21:51:42.573] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testLC0eYF/0 with map size 1073741824 21:51:42 [21:51:42.575] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.595] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:03705bacd2e68ba9 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:03705bacd2e68ba9"} 21:51:42 [21:51:42.596] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:edc5d0e4d384d88f epochNumber=0 {"provingJobId":"1:BASE_PARITY:edc5d0e4d384d88f"} 21:51:42 [21:51:42.597] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:03705bacd2e68ba9 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:03705bacd2e68ba9"} 21:51:42 [21:51:42.597] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d3464a5425730fbc epochNumber=0 {"provingJobId":"1:BASE_PARITY:d3464a5425730fbc"} 21:51:42 [21:51:42.615] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.617] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.618] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c346f836b56f27a7 epochNumber=2 {"provingJobId":"1:BASE_PARITY:c346f836b56f27a7"} 21:51:42 [21:51:42.618] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testeoPzAX/2 with map size 1073741824 21:51:42 [21:51:42.620] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.639] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b3fd1e5f0244e740 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b3fd1e5f0244e740"} 21:51:42 [21:51:42.640] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testeoPzAX/1 with map size 1073741824 21:51:42 [21:51:42.642] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.664] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.665] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fe7db70417d9bc13 epochNumber=1 {"provingJobId":"1:BASE_PARITY:fe7db70417d9bc13"} 21:51:42 [21:51:42.666] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwKM4Pg/1 with map size 1073741824 21:51:42 [21:51:42.668] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.687] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.688] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:90e572a4d7e4f47b epochNumber=1 {"provingJobId":"1:BASE_PARITY:90e572a4d7e4f47b"} 21:51:42 [21:51:42.689] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSQCzvX/1 with map size 1073741824 21:51:42 [21:51:42.690] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.706] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:78c32ed65bd35bd7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:78c32ed65bd35bd7"} 21:51:42 [21:51:42.712] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cf3a8375320d7318 epochNumber=2 {"provingJobId":"1:BASE_PARITY:cf3a8375320d7318"} 21:51:42 [21:51:42.712] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSQCzvX/2 with map size 1073741824 21:51:42 [21:51:42.714] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.732] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:57805ef6874fb00b epochNumber=1 {"provingJobId":"1:BASE_PARITY:57805ef6874fb00b"} 21:51:42 [21:51:42.742] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.743] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8e4519b777788ab3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8e4519b777788ab3"} 21:51:42 [21:51:42.744] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3eMANh/1 with map size 1073741824 21:51:42 [21:51:42.746] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.763] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:31e536a5315755ae epochNumber=1 {"provingJobId":"1:BASE_PARITY:31e536a5315755ae"} 21:51:42 [21:51:42.770] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:642972e6c9d6eb9c epochNumber=2 {"provingJobId":"1:BASE_PARITY:642972e6c9d6eb9c"} 21:51:42 [21:51:42.771] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test3eMANh/2 with map size 1073741824 21:51:42 [21:51:42.774] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.793] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2e7d5fa1b60a46a3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2e7d5fa1b60a46a3"} 21:51:42 [21:51:42.802] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.803] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:64f727546d8abdf6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:64f727546d8abdf6"} 21:51:42 [21:51:42.805] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxYrzzP/1 with map size 1073741824 21:51:42 [21:51:42.806] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.826] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5a25030d08667b5d epochNumber=1 {"provingJobId":"1:BASE_PARITY:5a25030d08667b5d"} 21:51:42 [21:51:42.834] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2a4d990fa58ce024 epochNumber=2 {"provingJobId":"1:BASE_PARITY:2a4d990fa58ce024"} 21:51:42 [21:51:42.835] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testxYrzzP/2 with map size 1073741824 21:51:42 [21:51:42.838] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.858] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dfb47c0317a55e2f epochNumber=1 {"provingJobId":"1:BASE_PARITY:dfb47c0317a55e2f"} 21:51:42 [21:51:42.868] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.869] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b7042666f391b5ec epochNumber=1 {"provingJobId":"1:BASE_PARITY:b7042666f391b5ec"} 21:51:42 [21:51:42.870] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBrqm2e/1 with map size 1073741824 21:51:42 [21:51:42.874] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.894] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:38a7b51fded35d1e epochNumber=1 {"provingJobId":"1:BASE_PARITY:38a7b51fded35d1e"} 21:51:42 [21:51:42.902] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b7042666f391b5ec type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b7042666f391b5ec"} 21:51:42 [21:51:42.913] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.914] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8bd5330104098160 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8bd5330104098160"} 21:51:42 [21:51:42.914] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testcD870b/1 with map size 1073741824 21:51:42 [21:51:42.916] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.934] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8b41f8cc802d9284 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8b41f8cc802d9284"} 21:51:42 [21:51:42.940] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8bd5330104098160 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:8bd5330104098160"} 21:51:42 [21:51:42.950] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.951] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0f62d0e28f3ef073 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0f62d0e28f3ef073"} 21:51:42 [21:51:42.954] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAirk7Q/1 with map size 1073741824 21:51:42 [21:51:42.956] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:42 [21:51:42.979] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6bd49aca0a8fc2da epochNumber=1 {"provingJobId":"1:BASE_PARITY:6bd49aca0a8fc2da"} 21:51:42 [21:51:42.986] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0f62d0e28f3ef073 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:0f62d0e28f3ef073"} 21:51:42 [21:51:42.992] INFO: prover-client:proving-broker Proving Broker started 21:51:42 [21:51:42.993] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:94a47644f7a4e5ce epochNumber=1 {"provingJobId":"1:BASE_PARITY:94a47644f7a4e5ce"} 21:51:42 [21:51:42.995] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgehPYI/1 with map size 1073741824 21:51:42 [21:51:42.997] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:43 [21:51:43.016] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:94a47644f7a4e5ce {"provingJobId":"1:BASE_PARITY:94a47644f7a4e5ce"} 21:51:43 [21:51:43.017] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:94a47644f7a4e5ce type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:94a47644f7a4e5ce"} 21:51:43 [21:51:43.024] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7f1ce0f438d2559b epochNumber=1 {"provingJobId":"1:BASE_PARITY:7f1ce0f438d2559b"} 21:51:43 [21:51:43.030] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:94a47644f7a4e5ce has already been completed {"provingJobId":"1:BASE_PARITY:94a47644f7a4e5ce"} 21:51:43 [21:51:43.035] INFO: prover-client:proving-broker Proving Broker started 21:51:43 [21:51:43.037] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dd4ca91b2409dcf9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dd4ca91b2409dcf9"} 21:51:43 [21:51:43.037] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6HFVpg/1 with map size 1073741824 21:51:43 [21:51:43.040] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:43 [21:51:43.057] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:463d20b5c68fca24 epochNumber=2 {"provingJobId":"1:BASE_PARITY:463d20b5c68fca24"} 21:51:43 [21:51:43.058] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6HFVpg/2 with map size 1073741824 21:51:43 [21:51:43.060] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:44 [21:51:44.131] INFO: prover-client:proving-broker Proving Broker started 21:51:44 [21:51:44.137] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:dd4ca91b2409dcf9 settled=false {"provingJobId":"1:BASE_PARITY:dd4ca91b2409dcf9","status":"pending"} 21:51:44 [21:51:44.139] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:463d20b5c68fca24 settled=false {"provingJobId":"1:BASE_PARITY:463d20b5c68fca24","status":"pending"} 21:51:44 [21:51:44.140] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:dd4ca91b2409dcf9 type=BASE_PARITY startedAt=1750110703080 older agent has taken job {"provingJobId":"1:BASE_PARITY:dd4ca91b2409dcf9"} 21:51:44 [21:51:44.140] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:dd4ca91b2409dcf9 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:dd4ca91b2409dcf9"} 21:51:44 [21:51:44.144] INFO: prover-client:proving-broker Proving Broker started 21:51:44 [21:51:44.145] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:139fa8d50f35fd4a epochNumber=1 {"provingJobId":"1:BASE_PARITY:139fa8d50f35fd4a"} 21:51:44 [21:51:44.146] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbnrSap/1 with map size 1073741824 21:51:44 [21:51:44.149] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:44 [21:51:44.168] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1747e487b31673a3 epochNumber=2 {"provingJobId":"1:BASE_PARITY:1747e487b31673a3"} 21:51:44 [21:51:44.175] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testbnrSap/2 with map size 1073741824 21:51:44 [21:51:44.178] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:45 [21:51:45.199] INFO: prover-client:proving-broker Proving Broker started 21:51:45 [21:51:45.201] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:139fa8d50f35fd4a settled=false {"provingJobId":"1:BASE_PARITY:139fa8d50f35fd4a","status":"pending"} 21:51:45 [21:51:45.202] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:1747e487b31673a3 settled=false {"provingJobId":"1:BASE_PARITY:1747e487b31673a3","status":"pending"} 21:51:45 [21:51:45.203] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:139fa8d50f35fd4a type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:139fa8d50f35fd4a"} 21:51:45 [21:51:45.206] INFO: prover-client:proving-broker Proving Broker started 21:51:45 [21:51:45.207] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0d8540bb20a090cf epochNumber=1 {"provingJobId":"1:BASE_PARITY:0d8540bb20a090cf"} 21:51:45 [21:51:45.208] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6NA43g/1 with map size 1073741824 21:51:45 [21:51:45.210] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:45 [21:51:45.227] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5b7e3846e749a2c5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:5b7e3846e749a2c5"} 21:51:45 [21:51:45.228] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6NA43g/2 with map size 1073741824 21:51:45 [21:51:45.230] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:46 [21:51:46.246] INFO: prover-client:proving-broker Proving Broker started 21:51:46 [21:51:46.248] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0d8540bb20a090cf settled=false {"provingJobId":"1:BASE_PARITY:0d8540bb20a090cf","status":"pending"} 21:51:46 [21:51:46.250] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5b7e3846e749a2c5 settled=false {"provingJobId":"1:BASE_PARITY:5b7e3846e749a2c5","status":"pending"} 21:51:46 [21:51:46.252] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0d8540bb20a090cf type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0d8540bb20a090cf"} 21:51:46 [21:51:46.252] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0d8540bb20a090cf type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0d8540bb20a090cf"} 21:51:46 [21:51:46.263] INFO: prover-client:proving-broker Proving Broker started 21:51:46 [21:51:46.264] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bb64fd9a58eef69e epochNumber=1 {"provingJobId":"1:BASE_PARITY:bb64fd9a58eef69e"} 21:51:46 [21:51:46.264] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testn2Cfc7/1 with map size 1073741824 21:51:46 [21:51:46.267] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:46 [21:51:46.289] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:89fbe0f0d590c878 epochNumber=2 {"provingJobId":"1:BASE_PARITY:89fbe0f0d590c878"} 21:51:46 [21:51:46.291] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testn2Cfc7/2 with map size 1073741824 21:51:46 [21:51:46.293] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:46 [21:51:46.328] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:bb64fd9a58eef69e type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:bb64fd9a58eef69e"} 21:51:46 [21:51:46.336] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:89fbe0f0d590c878 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:89fbe0f0d590c878"} 21:51:46 [21:51:46.348] INFO: prover-client:proving-broker Proving Broker started 21:51:46 [21:51:46.349] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c60a788b06f0b217 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c60a788b06f0b217"} 21:51:46 [21:51:46.352] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkuJyrO/1 with map size 1073741824 21:51:46 [21:51:46.354] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:46 [21:51:46.379] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5e1174ac5cead168 epochNumber=2 {"provingJobId":"1:BASE_PARITY:5e1174ac5cead168"} 21:51:46 [21:51:46.382] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkuJyrO/2 with map size 1073741824 21:51:46 [21:51:46.386] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:46 [21:51:46.409] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c60a788b06f0b217 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:c60a788b06f0b217"} 21:51:46 [21:51:46.410] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c60a788b06f0b217 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:c60a788b06f0b217"} 21:51:46 [21:51:46.416] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5e1174ac5cead168 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:5e1174ac5cead168"} 21:51:46 [21:51:46.427] INFO: prover-client:proving-broker Proving Broker started 21:51:46 [21:51:46.428] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:15cb9aae2803a83c err=test error {"provingJoId":"1:BASE_PARITY:15cb9aae2803a83c"} 21:51:46 [21:51:46.434] INFO: prover-client:proving-broker Proving Broker started 21:51:46 [21:51:46.435] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:704d7d22ee556d7d not found {"provingJobId":"1:BASE_PARITY:704d7d22ee556d7d"} 21:51:46 [21:51:46.440] INFO: prover-client:proving-broker Proving Broker started 21:51:46 [21:51:46.441] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0743788aa125b0da epochNumber=1 {"provingJobId":"1:BASE_PARITY:0743788aa125b0da"} 21:51:46 [21:51:46.442] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlK6pdd/1 with map size 1073741824 21:51:46 [21:51:46.445] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:46 [21:51:46.474] INFO: prover-client:proving-broker Proving Broker started 21:51:46 [21:51:46.475] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f58e6e25f43108b9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f58e6e25f43108b9"} 21:51:46 [21:51:46.478] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUFE60d/1 with map size 1073741824 21:51:46 [21:51:46.481] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:46 [21:51:46.627] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f58e6e25f43108b9 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:f58e6e25f43108b9"} 21:51:46 [21:51:46.634] INFO: prover-client:proving-broker Proving Broker started 21:51:46 [21:51:46.635] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d7cf1816b7f2bab3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d7cf1816b7f2bab3"} 21:51:46 [21:51:46.636] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFyingx/1 with map size 1073741824 21:51:46 [21:51:46.638] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:46 [21:51:46.760] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d7cf1816b7f2bab3 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:d7cf1816b7f2bab3"} 21:51:46 [21:51:46.762] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:06d12c2d847c8a87 epochNumber=10 {"provingJobId":"1:BASE_PARITY:06d12c2d847c8a87"} 21:51:46 [21:51:46.762] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testFyingx/10 with map size 1073741824 21:51:46 [21:51:46.768] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:46 [21:51:46.812] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:46 [21:51:46.813] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 21:51:46 [21:51:46.922] INFO: prover-client:proving-broker Proving Broker started 21:51:46 [21:51:46.923] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae51bfd21d8e8482 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ae51bfd21d8e8482"} 21:51:46 [21:51:46.924] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt1ElRN/1 with map size 1073741824 21:51:46 [21:51:46.926] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.125] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ae51bfd21d8e8482 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:ae51bfd21d8e8482"} 21:51:47 [21:51:47.156] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.157] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b0a84d3c1b6b8ea8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b0a84d3c1b6b8ea8"} 21:51:47 [21:51:47.158] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testV8Lk4p/1 with map size 1073741824 21:51:47 [21:51:47.161] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.180] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:b0a84d3c1b6b8ea8 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:b0a84d3c1b6b8ea8"} 21:51:47 [21:51:47.186] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.187] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4ec8dda176141044 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4ec8dda176141044"} 21:51:47 [21:51:47.188] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4p7PCH/1 with map size 1073741824 21:51:47 [21:51:47.192] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.215] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:4ec8dda176141044 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:4ec8dda176141044"} 21:51:47 [21:51:47.216] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4ec8dda176141044 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:4ec8dda176141044"} 21:51:47 [21:51:47.229] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.230] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7913e2dde45a8639 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7913e2dde45a8639"} 21:51:47 [21:51:47.233] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAALIUK/1 with map size 1073741824 21:51:47 [21:51:47.236] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.259] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7913e2dde45a8639 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:7913e2dde45a8639"} 21:51:47 [21:51:47.269] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.271] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d7cc915ebb53714a epochNumber=1 {"provingJobId":"1:BASE_PARITY:d7cc915ebb53714a"} 21:51:47 [21:51:47.272] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTwZxbB/1 with map size 1073741824 21:51:47 [21:51:47.275] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.297] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d7cc915ebb53714a type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:d7cc915ebb53714a"} 21:51:47 [21:51:47.298] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a14eacefdaf4109b epochNumber=3 {"provingJobId":"1:BASE_PARITY:a14eacefdaf4109b"} 21:51:47 [21:51:47.301] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testTwZxbB/3 with map size 1073741824 21:51:47 [21:51:47.303] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.345] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:47 [21:51:47.346] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 21:51:47 [21:51:47.351] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d7cc915ebb53714a err=test error {"provingJoId":"1:BASE_PARITY:d7cc915ebb53714a"} 21:51:47 [21:51:47.356] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmRE9vW/1 with map size 1073741824 21:51:47 [21:51:47.358] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.379] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testmRE9vW/2 with map size 1073741824 21:51:47 [21:51:47.382] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.403] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.405] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6c8a900b3612897a settled=false {"provingJobId":"1:BASE_PARITY:6c8a900b3612897a","status":"pending"} 21:51:47 [21:51:47.406] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6911ca185f962132 settled=false {"provingJobId":"1:BASE_PARITY:6911ca185f962132","status":"pending"} 21:51:47 [21:51:47.415] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testC98DJo/1 with map size 1073741824 21:51:47 [21:51:47.418] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.442] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testC98DJo/2 with map size 1073741824 21:51:47 [21:51:47.444] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.472] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.475] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5080109c47155365 settled=true {"provingJobId":"1:BASE_PARITY:5080109c47155365","status":"fulfilled"} 21:51:47 [21:51:47.477] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:b408699c8c820440 settled=true {"provingJobId":"2:BASE_PARITY:b408699c8c820440","status":"fulfilled"} 21:51:47 [21:51:47.486] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testCqkkZk/1 with map size 1073741824 21:51:47 [21:51:47.489] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.513] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testCqkkZk/2 with map size 1073741824 21:51:47 [21:51:47.517] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.535] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.537] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d441a13904a03644 settled=true {"provingJobId":"1:BASE_PARITY:d441a13904a03644","status":"fulfilled"} 21:51:47 [21:51:47.539] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5bb128e20c6bb6c2 settled=false {"provingJobId":"1:BASE_PARITY:5bb128e20c6bb6c2","status":"pending"} 21:51:47 [21:51:47.545] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:32516789145b7513 epochNumber=1 {"provingJobId":"1:BASE_PARITY:32516789145b7513"} 21:51:47 [21:51:47.548] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpe3TE8/1 with map size 1073741824 21:51:47 [21:51:47.550] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.577] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.578] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:35020ca2eb0a031f epochNumber=1 {"provingJobId":"1:BASE_PARITY:35020ca2eb0a031f"} 21:51:47 [21:51:47.579] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:35020ca2eb0a031f: Error: db error: Error: db error 21:51:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:35020ca2eb0a031f"} 21:51:47 [21:51:47.584] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.585] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:27aceab942d4c6ce epochNumber=1 {"provingJobId":"1:BASE_PARITY:27aceab942d4c6ce"} 21:51:47 [21:51:47.586] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbXyTYa/1 with map size 1073741824 21:51:47 [21:51:47.588] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.607] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:27aceab942d4c6ce type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:27aceab942d4c6ce"} 21:51:47 [21:51:47.607] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:27aceab942d4c6ce type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:27aceab942d4c6ce"} 21:51:47 [21:51:47.617] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.617] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:78574d52a3db52e7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:78574d52a3db52e7"} 21:51:47 [21:51:47.620] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testaVSynh/1 with map size 1073741824 21:51:47 [21:51:47.622] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.646] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:78574d52a3db52e7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:78574d52a3db52e7"} 21:51:47 [21:51:47.647] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:78574d52a3db52e7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:78574d52a3db52e7"} 21:51:47 [21:51:47.647] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:78574d52a3db52e7: Error: db error 21:51:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:78574d52a3db52e7"} 21:51:47 [21:51:47.652] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.653] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cd8ba89d5346f7f2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cd8ba89d5346f7f2"} 21:51:47 [21:51:47.656] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlOqyaA/1 with map size 1073741824 21:51:47 [21:51:47.662] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.691] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cd8ba89d5346f7f2 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cd8ba89d5346f7f2"} 21:51:47 [21:51:47.691] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:cd8ba89d5346f7f2 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:cd8ba89d5346f7f2"} 21:51:47 [21:51:47.701] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.701] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f1ebc1783138eec8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f1ebc1783138eec8"} 21:51:47 [21:51:47.702] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjmEnw7/1 with map size 1073741824 21:51:47 [21:51:47.707] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.727] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f1ebc1783138eec8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f1ebc1783138eec8"} 21:51:47 [21:51:47.728] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f1ebc1783138eec8 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:f1ebc1783138eec8"} 21:51:47 [21:51:47.729] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:f1ebc1783138eec8 jobErr=test error: Error: db error 21:51:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:f1ebc1783138eec8"} 21:51:47 [21:51:47.738] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.739] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7e7874bf709410c1 not found {"provingJobId":"1:BASE_PARITY:7e7874bf709410c1"} 21:51:47 [21:51:47.744] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.745] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0cc7d2e9cd280d9d err=test error {"provingJoId":"1:BASE_PARITY:0cc7d2e9cd280d9d"} 21:51:47 [21:51:47.748] INFO: prover-client:proving-broker Proving Broker started 21:51:47 [21:51:47.758] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7233c4885d91b255 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7233c4885d91b255"} 21:51:47 [21:51:47.760] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testop64AH/1 with map size 1073741824 21:51:47 [21:51:47.770] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.793] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7233c4885d91b255 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:7233c4885d91b255"} 21:51:47 [21:51:47.793] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7233c4885d91b255 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:7233c4885d91b255"} 21:51:47 [21:51:47.810] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:86c6d3e8f5db5a5f epochNumber=2 {"provingJobId":"2:BASE_PARITY:86c6d3e8f5db5a5f"} 21:51:47 [21:51:47.811] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testop64AH/2 with map size 1073741824 21:51:47 [21:51:47.814] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.833] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:86c6d3e8f5db5a5f type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:86c6d3e8f5db5a5f"} 21:51:47 [21:51:47.834] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:86c6d3e8f5db5a5f type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:86c6d3e8f5db5a5f"} 21:51:47 [21:51:47.902] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:fb1c47089d758170 epochNumber=3 {"provingJobId":"3:BASE_PARITY:fb1c47089d758170"} 21:51:47 [21:51:47.903] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testop64AH/3 with map size 1073741824 21:51:47 [21:51:47.909] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:47 [21:51:47.931] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:47 [21:51:47.932] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 21:51:47 [21:51:47.990] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:6543415eb3b3f023 epochNumber=4 {"provingJobId":"4:BASE_PARITY:6543415eb3b3f023"} 21:51:47 [21:51:47.991] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testop64AH/4 with map size 1073741824 21:51:47 [21:51:47.996] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:48 [21:51:48.039] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:48 [21:51:48.039] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 21:51:48 [21:51:48.076] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:1337b618e51d9118 epochNumber=5 {"provingJobId":"5:BASE_PARITY:1337b618e51d9118"} 21:51:48 [21:51:48.078] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testop64AH/5 with map size 1073741824 21:51:48 [21:51:48.080] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 21:51:48 [21:51:48.119] INFO: prover-client:proving-broker Cleaned up jobs=1 21:51:48 [21:51:48.120] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 21:51:48 PASS src/proving_broker/proving_broker.test.ts (13.49 s) 21:51:48 ProvingBroker 21:51:48 Producer API 21:51:48 ✓ refuses stale jobs (51 ms) 21:51:48 ✓ enqueues jobs (6 ms) 21:51:48 ✓ ignores duplicate jobs (6 ms) 21:51:48 ✓ reports correct status when enqueuing repeat jobs (7 ms) 21:51:48 ✓ throws an error in case of duplicate job IDs (45 ms) 21:51:48 ✓ returns not-found status for non-existing jobs (3 ms) 21:51:48 ✓ cancels jobs in queue (4 ms) 21:51:48 ✓ cancels jobs in-progress (3 ms) 21:51:48 ✓ returns job result if successful (4 ms) 21:51:48 ✓ returns job error if failed (3 ms) 21:51:48 ✓ correctly returns job status for concurrent writes (3 ms) 21:51:48 ✓ refuses stale jobs (45 ms) 21:51:48 ✓ enqueues jobs (37 ms) 21:51:48 ✓ ignores duplicate jobs (27 ms) 21:51:48 ✓ reports correct status when enqueuing repeat jobs (40 ms) 21:51:48 ✓ throws an error in case of duplicate job IDs (28 ms) 21:51:48 ✓ returns not-found status for non-existing jobs (3 ms) 21:51:48 ✓ cancels jobs in queue (33 ms) 21:51:48 ✓ cancels jobs in-progress (36 ms) 21:51:48 ✓ returns job result if successful (32 ms) 21:51:48 ✓ returns job error if failed (28 ms) 21:51:48 ✓ correctly returns job status for concurrent writes (43 ms) 21:51:48 Consumer API 21:51:48 ✓ returns undefined if no jobs are available (1 ms) 21:51:48 ✓ returns jobs in priority order (2 ms) 21:51:48 ✓ returns undefined if no jobs are available for the given allowList (1 ms) 21:51:48 ✓ returns a job if it is in the allowList (3 ms) 21:51:48 ✓ returns the most important job if it is in the allowList (3 ms) 21:51:48 ✓ returns any job if filter is empty (5 ms) 21:51:48 ✓ returns a new job when reporting job success (4 ms) 21:51:48 ✓ returns a new job when reporting permanent error (3 ms) 21:51:48 ✓ returns a new job when reporting retry-able error (2 ms) 21:51:48 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 21:51:48 ✓ returns a new job if job is already in progress elsewhere (1059 ms) 21:51:48 ✓ avoids sending the same job to a new agent after a restart (1005 ms) 21:51:48 ✓ avoids sending a completed job to a new agent after a restart (1005 ms) 21:51:48 ✓ tracks job result if in progress (2 ms) 21:51:48 ✓ tracks job result even if job is in queue (3 ms) 21:51:48 ✓ ignores reported job error if unknown job (1 ms) 21:51:48 ✓ ignores job result if unknown job (1 ms) 21:51:48 ✓ returns undefined if no jobs are available (2 ms) 21:51:48 ✓ returns jobs in priority order (46 ms) 21:51:48 ✓ returns undefined if no jobs are available for the given allowList (24 ms) 21:51:48 ✓ returns a job if it is in the allowList (54 ms) 21:51:48 ✓ returns the most important job if it is in the allowList (60 ms) 21:51:48 ✓ returns any job if filter is empty (65 ms) 21:51:48 ✓ returns a new job when reporting job success (44 ms) 21:51:48 ✓ returns a new job when reporting permanent error (36 ms) 21:51:48 ✓ returns a new job when reporting retry-able error (41 ms) 21:51:48 ✓ returns a new job when reporting progress if current one is cancelled (42 ms) 21:51:48 ✓ returns a new job if job is already in progress elsewhere (1109 ms) 21:51:48 ✓ avoids sending the same job to a new agent after a restart (1062 ms) 21:51:48 ✓ avoids sending a completed job to a new agent after a restart (1055 ms) 21:51:48 ✓ tracks job result if in progress (84 ms) 21:51:48 ✓ tracks job result even if job is in queue (78 ms) 21:51:48 ✓ ignores reported job error if unknown job (5 ms) 21:51:48 ✓ ignores job result if unknown job (7 ms) 21:51:48 Timeouts 21:51:48 ✓ tracks in progress jobs (2 ms) 21:51:48 ✓ re-enqueues jobs that time out (130 ms) 21:51:48 ✓ cancel stale jobs that time out (236 ms) 21:51:48 ✓ keeps the jobs in progress while it is alive (205 ms) 21:51:48 ✓ tracks in progress jobs (30 ms) 21:51:48 ✓ re-enqueues jobs that time out (164 ms) 21:51:48 ✓ cancel stale jobs that time out (287 ms) 21:51:48 ✓ keeps the jobs in progress while it is alive (234 ms) 21:51:48 Retries 21:51:48 ✓ retries jobs (3 ms) 21:51:48 ✓ retries up to a maximum number of times (3 ms) 21:51:48 ✓ passing retry=false does not retry (2 ms) 21:51:48 ✓ does not retry if job is stale (29 ms) 21:51:48 ✓ retries jobs (28 ms) 21:51:48 ✓ retries up to a maximum number of times (44 ms) 21:51:48 ✓ passing retry=false does not retry (40 ms) 21:51:48 ✓ does not retry if job is stale (86 ms) 21:51:48 Database management 21:51:48 ✓ re-enqueues proof requests on start (3 ms) 21:51:48 ✓ restores proof results on start (2 ms) 21:51:48 ✓ only re-enqueues unfinished jobs (2 ms) 21:51:48 ✓ saves job when enqueued (3 ms) 21:51:48 ✓ does not retain job if database fails to save (3 ms) 21:51:48 ✓ saves job result (2 ms) 21:51:48 ✓ saves result even if database fails to save (2 ms) 21:51:48 ✓ saves job error (1 ms) 21:51:48 ✓ saves job error even if database fails to save (1 ms) 21:51:48 ✓ does not save job result if job is unknown (1 ms) 21:51:48 ✓ does not save job error if job is unknown (1 ms) 21:51:48 ✓ cleans up old jobs periodically (261 ms) 21:51:48 ✓ re-enqueues proof requests on start (57 ms) 21:51:48 ✓ restores proof results on start (69 ms) 21:51:48 ✓ only re-enqueues unfinished jobs (62 ms) 21:51:48 ✓ saves job when enqueued (31 ms) 21:51:48 ✓ does not retain job if database fails to save (8 ms) 21:51:48 ✓ saves job result (32 ms) 21:51:48 ✓ saves result even if database fails to save (34 ms) 21:51:48 ✓ saves job error (49 ms) 21:51:48 ✓ saves job error even if database fails to save (34 ms) 21:51:48 ✓ does not save job result if job is unknown (8 ms) 21:51:48 ✓ does not save job error if job is unknown (5 ms) 21:51:48 ✓ cleans up old jobs periodically (412 ms) 21:51:48 21:51:48 Test Suites: 1 passed, 1 total 21:51:48 Tests: 96 passed, 96 total 21:51:48 Snapshots: 0 total 21:51:48 Time: 13.604 s 21:51:48 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 21:51:48 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?