Parent Log: http://ci.aztec-labs.com/9c875b57c7229e8a Command: bdef8c432fe28910 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/a9f91dd1e5946aecfeb192ce869eb3e8eca2742e Env: REF_NAME=gh-readonly-queue/next/pr-15099-a48851a8c6884f4b9615a154f8862a549722d25e CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Tue Jun 17 12:33:56 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15099_amd64_x2-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next 12:34:00 [12:34:00.816] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.825] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:05a928d7fb3559c8 epochNumber=42 {"provingJobId":"1:BASE_PARITY:05a928d7fb3559c8"} 12:34:00 [12:34:00.831] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:1aff70188d563bb7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1aff70188d563bb7"} 12:34:00 [12:34:00.872] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.873] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5addd081728fe118 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5addd081728fe118"} 12:34:00 [12:34:00.875] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a1c0b92c12cb231f epochNumber=1 {"provingJobId":"1:BASE_PARITY:a1c0b92c12cb231f"} 12:34:00 [12:34:00.877] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.878] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d18970a26830f956 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d18970a26830f956"} 12:34:00 [12:34:00.879] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d18970a26830f956 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:d18970a26830f956"} 12:34:00 [12:34:00.881] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.882] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c4cde2b4203a3c8a epochNumber=1 {"provingJobId":"1:BASE_PARITY:c4cde2b4203a3c8a"} 12:34:00 [12:34:00.883] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c4cde2b4203a3c8a epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:c4cde2b4203a3c8a"} 12:34:00 [12:34:00.883] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:c4cde2b4203a3c8a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:c4cde2b4203a3c8a"} 12:34:00 [12:34:00.884] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:c4cde2b4203a3c8a epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:c4cde2b4203a3c8a"} 12:34:00 [12:34:00.886] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.887] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4b5acc3bd5896add epochNumber=1 {"provingJobId":"1:BASE_PARITY:4b5acc3bd5896add"} 12:34:00 [12:34:00.922] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.923] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.924] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b45fb8357d3aecf4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b45fb8357d3aecf4"} 12:34:00 [12:34:00.925] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:b45fb8357d3aecf4 {"provingJobId":"1:BASE_PARITY:b45fb8357d3aecf4"} 12:34:00 [12:34:00.925] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b45fb8357d3aecf4 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b45fb8357d3aecf4"} 12:34:00 [12:34:00.925] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b45fb8357d3aecf4 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:b45fb8357d3aecf4"} 12:34:00 [12:34:00.927] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.927] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:99eb15f7b7ae79e1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:99eb15f7b7ae79e1"} 12:34:00 [12:34:00.928] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:99eb15f7b7ae79e1 {"provingJobId":"1:BASE_PARITY:99eb15f7b7ae79e1"} 12:34:00 [12:34:00.929] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:99eb15f7b7ae79e1 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:99eb15f7b7ae79e1"} 12:34:00 [12:34:00.930] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.931] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f0f7ab1484e9799a epochNumber=1 {"provingJobId":"1:BASE_PARITY:f0f7ab1484e9799a"} 12:34:00 [12:34:00.942] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:f0f7ab1484e9799a type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:f0f7ab1484e9799a"} 12:34:00 [12:34:00.942] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:f0f7ab1484e9799a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:f0f7ab1484e9799a"} 12:34:00 [12:34:00.947] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.948] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6be71214387f77ef epochNumber=1 {"provingJobId":"1:BASE_PARITY:6be71214387f77ef"} 12:34:00 [12:34:00.949] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6be71214387f77ef type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:6be71214387f77ef"} 12:34:00 [12:34:00.949] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6be71214387f77ef type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:6be71214387f77ef"} 12:34:00 [12:34:00.953] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.953] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b3bce95354117cd4 epochNumber=0 {"provingJobId":"1:BASE_PARITY:b3bce95354117cd4"} 12:34:00 [12:34:00.954] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b3bce95354117cd4 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b3bce95354117cd4"} 12:34:00 [12:34:00.954] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:76cae6f0c08eda84 epochNumber=0 {"provingJobId":"1:BASE_PARITY:76cae6f0c08eda84"} 12:34:00 [12:34:00.954] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b3bce95354117cd4 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:b3bce95354117cd4"} 12:34:00 [12:34:00.954] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e3b13d069dfe14ce epochNumber=0 {"provingJobId":"1:BASE_PARITY:e3b13d069dfe14ce"} 12:34:00 [12:34:00.956] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.958] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.958] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:93d83ff808d31561 epochNumber=2 {"provingJobId":"1:BASE_PARITY:93d83ff808d31561"} 12:34:00 [12:34:00.959] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2d0ced64d9e08d5d epochNumber=1 {"provingJobId":"1:BASE_PARITY:2d0ced64d9e08d5d"} 12:34:00 [12:34:00.961] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.962] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a3f3812d3889c5bc epochNumber=1 {"provingJobId":"1:BASE_PARITY:a3f3812d3889c5bc"} 12:34:00 [12:34:00.963] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.964] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ef519c6fb5449676 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ef519c6fb5449676"} 12:34:00 [12:34:00.964] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:99d233129cc00f49 epochNumber=1 {"provingJobId":"1:BASE_PARITY:99d233129cc00f49"} 12:34:00 [12:34:00.964] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7a5a516a465ca652 epochNumber=2 {"provingJobId":"1:BASE_PARITY:7a5a516a465ca652"} 12:34:00 [12:34:00.965] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fdd5159ed88037ac epochNumber=1 {"provingJobId":"1:BASE_PARITY:fdd5159ed88037ac"} 12:34:00 [12:34:00.966] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.966] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:776ffca54e472e99 epochNumber=1 {"provingJobId":"1:BASE_PARITY:776ffca54e472e99"} 12:34:00 [12:34:00.967] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:008604fbdb9fdcfc epochNumber=1 {"provingJobId":"1:BASE_PARITY:008604fbdb9fdcfc"} 12:34:00 [12:34:00.967] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3077e5f8083bc563 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3077e5f8083bc563"} 12:34:00 [12:34:00.967] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7063d034ee08d297 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7063d034ee08d297"} 12:34:00 [12:34:00.969] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.969] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4a28aebd7e90452b epochNumber=1 {"provingJobId":"1:BASE_PARITY:4a28aebd7e90452b"} 12:34:00 [12:34:00.970] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e2a5c580d1e3adcc epochNumber=1 {"provingJobId":"1:BASE_PARITY:e2a5c580d1e3adcc"} 12:34:00 [12:34:00.970] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6a0089a40ca012c4 epochNumber=2 {"provingJobId":"1:BASE_PARITY:6a0089a40ca012c4"} 12:34:00 [12:34:00.970] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eaf8c6875d57736b epochNumber=1 {"provingJobId":"1:BASE_PARITY:eaf8c6875d57736b"} 12:34:00 [12:34:00.972] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.972] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b88cc0608a0c413c epochNumber=1 {"provingJobId":"1:BASE_PARITY:b88cc0608a0c413c"} 12:34:00 [12:34:00.973] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f530b6b5bb493f82 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f530b6b5bb493f82"} 12:34:00 [12:34:00.973] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b88cc0608a0c413c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b88cc0608a0c413c"} 12:34:00 [12:34:00.974] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.975] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d669dd61d02f4b8a epochNumber=1 {"provingJobId":"1:BASE_PARITY:d669dd61d02f4b8a"} 12:34:00 [12:34:00.975] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7d90e0bae1ba9a79 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7d90e0bae1ba9a79"} 12:34:00 [12:34:00.975] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d669dd61d02f4b8a type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:d669dd61d02f4b8a"} 12:34:00 [12:34:00.976] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.977] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:34dc5c6e2d11ce75 epochNumber=1 {"provingJobId":"1:BASE_PARITY:34dc5c6e2d11ce75"} 12:34:00 [12:34:00.977] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:432620e6a096190d epochNumber=1 {"provingJobId":"1:BASE_PARITY:432620e6a096190d"} 12:34:00 [12:34:00.977] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:34dc5c6e2d11ce75 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:34dc5c6e2d11ce75"} 12:34:00 [12:34:00.978] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.979] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e9f6a80930bc39f5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e9f6a80930bc39f5"} 12:34:00 [12:34:00.979] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:e9f6a80930bc39f5 {"provingJobId":"1:BASE_PARITY:e9f6a80930bc39f5"} 12:34:00 [12:34:00.979] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e9f6a80930bc39f5 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:e9f6a80930bc39f5"} 12:34:00 [12:34:00.980] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0180a4f8d07a19bf epochNumber=1 {"provingJobId":"1:BASE_PARITY:0180a4f8d07a19bf"} 12:34:00 [12:34:00.980] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e9f6a80930bc39f5 has already been completed {"provingJobId":"1:BASE_PARITY:e9f6a80930bc39f5"} 12:34:00 [12:34:00.981] INFO: prover-client:proving-broker Proving Broker started 12:34:00 [12:34:00.982] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0b679441e6ddbd99 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0b679441e6ddbd99"} 12:34:00 [12:34:00.982] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c1cddac2b9aa376c epochNumber=2 {"provingJobId":"1:BASE_PARITY:c1cddac2b9aa376c"} 12:34:02 [12:34:02.035] INFO: prover-client:proving-broker Proving Broker started 12:34:02 [12:34:02.036] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0b679441e6ddbd99 settled=false {"provingJobId":"1:BASE_PARITY:0b679441e6ddbd99","status":"pending"} 12:34:02 [12:34:02.036] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c1cddac2b9aa376c settled=false {"provingJobId":"1:BASE_PARITY:c1cddac2b9aa376c","status":"pending"} 12:34:02 [12:34:02.037] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:0b679441e6ddbd99 type=BASE_PARITY startedAt=1750163640983 older agent has taken job {"provingJobId":"1:BASE_PARITY:0b679441e6ddbd99"} 12:34:02 [12:34:02.038] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0b679441e6ddbd99 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:0b679441e6ddbd99"} 12:34:02 [12:34:02.039] INFO: prover-client:proving-broker Proving Broker started 12:34:02 [12:34:02.039] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ad0685701f3a56fe epochNumber=1 {"provingJobId":"1:BASE_PARITY:ad0685701f3a56fe"} 12:34:02 [12:34:02.039] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6f02d63e575c738e epochNumber=2 {"provingJobId":"1:BASE_PARITY:6f02d63e575c738e"} 12:34:03 [12:34:03.040] INFO: prover-client:proving-broker Proving Broker started 12:34:03 [12:34:03.040] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ad0685701f3a56fe settled=false {"provingJobId":"1:BASE_PARITY:ad0685701f3a56fe","status":"pending"} 12:34:03 [12:34:03.041] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6f02d63e575c738e settled=false {"provingJobId":"1:BASE_PARITY:6f02d63e575c738e","status":"pending"} 12:34:03 [12:34:03.042] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ad0685701f3a56fe type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:ad0685701f3a56fe"} 12:34:03 [12:34:03.043] INFO: prover-client:proving-broker Proving Broker started 12:34:03 [12:34:03.044] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e2589fa76e6ebbf9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e2589fa76e6ebbf9"} 12:34:03 [12:34:03.044] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:775b5842ba782c53 epochNumber=2 {"provingJobId":"1:BASE_PARITY:775b5842ba782c53"} 12:34:04 [12:34:04.045] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.045] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e2589fa76e6ebbf9 settled=false {"provingJobId":"1:BASE_PARITY:e2589fa76e6ebbf9","status":"pending"} 12:34:04 [12:34:04.046] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:775b5842ba782c53 settled=false {"provingJobId":"1:BASE_PARITY:775b5842ba782c53","status":"pending"} 12:34:04 [12:34:04.046] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e2589fa76e6ebbf9 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:e2589fa76e6ebbf9"} 12:34:04 [12:34:04.047] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e2589fa76e6ebbf9 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:e2589fa76e6ebbf9"} 12:34:04 [12:34:04.048] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.049] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:01fbb34db7e68679 epochNumber=1 {"provingJobId":"1:BASE_PARITY:01fbb34db7e68679"} 12:34:04 [12:34:04.049] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:478b7b43f5f1e5dc epochNumber=2 {"provingJobId":"1:BASE_PARITY:478b7b43f5f1e5dc"} 12:34:04 [12:34:04.050] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:01fbb34db7e68679 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:01fbb34db7e68679"} 12:34:04 [12:34:04.050] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:478b7b43f5f1e5dc type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:478b7b43f5f1e5dc"} 12:34:04 [12:34:04.051] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.051] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0a98f00ae3849201 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0a98f00ae3849201"} 12:34:04 [12:34:04.051] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fe7dfd00890c5c06 epochNumber=2 {"provingJobId":"1:BASE_PARITY:fe7dfd00890c5c06"} 12:34:04 [12:34:04.052] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0a98f00ae3849201 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0a98f00ae3849201"} 12:34:04 [12:34:04.052] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0a98f00ae3849201 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0a98f00ae3849201"} 12:34:04 [12:34:04.052] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fe7dfd00890c5c06 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:fe7dfd00890c5c06"} 12:34:04 [12:34:04.053] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.053] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c9dfd26931901047 err=test error {"provingJoId":"1:BASE_PARITY:c9dfd26931901047"} 12:34:04 [12:34:04.054] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.054] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:555dc2e01625766e not found {"provingJobId":"1:BASE_PARITY:555dc2e01625766e"} 12:34:04 [12:34:04.055] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.055] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8a7104f31c6257e6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8a7104f31c6257e6"} 12:34:04 [12:34:04.056] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.057] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:41579be46ab0bc33 epochNumber=1 {"provingJobId":"1:BASE_PARITY:41579be46ab0bc33"} 12:34:04 [12:34:04.158] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:41579be46ab0bc33 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:41579be46ab0bc33"} 12:34:04 [12:34:04.166] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.166] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:60c82ce70a0b1f9a epochNumber=1 {"provingJobId":"1:BASE_PARITY:60c82ce70a0b1f9a"} 12:34:04 [12:34:04.267] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:60c82ce70a0b1f9a timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:60c82ce70a0b1f9a"} 12:34:04 [12:34:04.269] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9493c91f6e91ed82 epochNumber=10 {"provingJobId":"1:BASE_PARITY:9493c91f6e91ed82"} 12:34:04 [12:34:04.293] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:04 [12:34:04.396] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.397] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae623fdb95e13504 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ae623fdb95e13504"} 12:34:04 [12:34:04.550] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ae623fdb95e13504 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:ae623fdb95e13504"} 12:34:04 [12:34:04.601] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.601] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a17e862db6e48e8b epochNumber=1 {"provingJobId":"1:BASE_PARITY:a17e862db6e48e8b"} 12:34:04 [12:34:04.602] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:a17e862db6e48e8b type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:a17e862db6e48e8b"} 12:34:04 [12:34:04.604] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.604] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8ed6ada8bcfedbb2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8ed6ada8bcfedbb2"} 12:34:04 [12:34:04.605] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:8ed6ada8bcfedbb2 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:8ed6ada8bcfedbb2"} 12:34:04 [12:34:04.606] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8ed6ada8bcfedbb2 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:8ed6ada8bcfedbb2"} 12:34:04 [12:34:04.607] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.607] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:75eba05cafd013aa epochNumber=1 {"provingJobId":"1:BASE_PARITY:75eba05cafd013aa"} 12:34:04 [12:34:04.608] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:75eba05cafd013aa type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:75eba05cafd013aa"} 12:34:04 [12:34:04.609] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.609] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0b7bfa326baf64b3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0b7bfa326baf64b3"} 12:34:04 [12:34:04.610] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0b7bfa326baf64b3 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:0b7bfa326baf64b3"} 12:34:04 [12:34:04.610] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2db2788ef5eb2f3e epochNumber=3 {"provingJobId":"1:BASE_PARITY:2db2788ef5eb2f3e"} 12:34:04 [12:34:04.633] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:04 [12:34:04.636] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:0b7bfa326baf64b3 err=test error {"provingJoId":"1:BASE_PARITY:0b7bfa326baf64b3"} 12:34:04 [12:34:04.639] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.639] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d0cf8f19a7523e9b settled=false {"provingJobId":"1:BASE_PARITY:d0cf8f19a7523e9b","status":"pending"} 12:34:04 [12:34:04.640] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:13d98ed0946f8422 settled=false {"provingJobId":"1:BASE_PARITY:13d98ed0946f8422","status":"pending"} 12:34:04 [12:34:04.642] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.642] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6793896f18580e0d settled=true {"provingJobId":"1:BASE_PARITY:6793896f18580e0d","status":"fulfilled"} 12:34:04 [12:34:04.642] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:f6de9a720f3f188b settled=true {"provingJobId":"2:BASE_PARITY:f6de9a720f3f188b","status":"fulfilled"} 12:34:04 [12:34:04.644] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.644] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d6a8e7f5f399ac16 settled=true {"provingJobId":"1:BASE_PARITY:d6a8e7f5f399ac16","status":"fulfilled"} 12:34:04 [12:34:04.644] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4ff05d2a25d87d95 settled=false {"provingJobId":"1:BASE_PARITY:4ff05d2a25d87d95","status":"pending"} 12:34:04 [12:34:04.645] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.646] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e585acf64a4271c6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e585acf64a4271c6"} 12:34:04 [12:34:04.647] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.648] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e00e9e039d03f581 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e00e9e039d03f581"} 12:34:04 [12:34:04.648] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:e00e9e039d03f581: Error: db error: Error: db error 12:34:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:e00e9e039d03f581"} 12:34:04 [12:34:04.649] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.649] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b7ba4ac216bcf8cc epochNumber=1 {"provingJobId":"1:BASE_PARITY:b7ba4ac216bcf8cc"} 12:34:04 [12:34:04.650] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b7ba4ac216bcf8cc type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b7ba4ac216bcf8cc"} 12:34:04 [12:34:04.650] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b7ba4ac216bcf8cc type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b7ba4ac216bcf8cc"} 12:34:04 [12:34:04.651] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.651] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ae1c87e712a1ac34 epochNumber=1 {"provingJobId":"1:BASE_PARITY:ae1c87e712a1ac34"} 12:34:04 [12:34:04.651] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ae1c87e712a1ac34 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ae1c87e712a1ac34"} 12:34:04 [12:34:04.651] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ae1c87e712a1ac34 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ae1c87e712a1ac34"} 12:34:04 [12:34:04.652] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:ae1c87e712a1ac34: Error: db error 12:34:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:ae1c87e712a1ac34"} 12:34:04 [12:34:04.653] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.653] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a509b6fbb6065503 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a509b6fbb6065503"} 12:34:04 [12:34:04.654] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a509b6fbb6065503 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a509b6fbb6065503"} 12:34:04 [12:34:04.654] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a509b6fbb6065503 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:a509b6fbb6065503"} 12:34:04 [12:34:04.655] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.655] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a0b68a7fbc5ccea8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a0b68a7fbc5ccea8"} 12:34:04 [12:34:04.656] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:a0b68a7fbc5ccea8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:a0b68a7fbc5ccea8"} 12:34:04 [12:34:04.656] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a0b68a7fbc5ccea8 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:a0b68a7fbc5ccea8"} 12:34:04 [12:34:04.657] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a0b68a7fbc5ccea8 jobErr=test error: Error: db error 12:34:04 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:a0b68a7fbc5ccea8"} 12:34:04 [12:34:04.663] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.664] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5b514cba6aee90ff not found {"provingJobId":"1:BASE_PARITY:5b514cba6aee90ff"} 12:34:04 [12:34:04.666] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.667] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:d0f2e9e5e74086f7 err=test error {"provingJoId":"1:BASE_PARITY:d0f2e9e5e74086f7"} 12:34:04 [12:34:04.668] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.679] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ef08a9103a43312d epochNumber=1 {"provingJobId":"1:BASE_PARITY:ef08a9103a43312d"} 12:34:04 [12:34:04.680] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:ef08a9103a43312d type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:ef08a9103a43312d"} 12:34:04 [12:34:04.680] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ef08a9103a43312d type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:ef08a9103a43312d"} 12:34:04 [12:34:04.691] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:e70c062f4a45dea9 epochNumber=2 {"provingJobId":"2:BASE_PARITY:e70c062f4a45dea9"} 12:34:04 [12:34:04.692] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:e70c062f4a45dea9 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:e70c062f4a45dea9"} 12:34:04 [12:34:04.692] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:e70c062f4a45dea9 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:e70c062f4a45dea9"} 12:34:04 [12:34:04.753] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:03078b9131b378a9 epochNumber=3 {"provingJobId":"3:BASE_PARITY:03078b9131b378a9"} 12:34:04 [12:34:04.773] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:04 [12:34:04.815] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:962bdb37b1034f17 epochNumber=4 {"provingJobId":"4:BASE_PARITY:962bdb37b1034f17"} 12:34:04 [12:34:04.824] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:04 [12:34:04.877] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:f8cf0931cae6600f epochNumber=5 {"provingJobId":"5:BASE_PARITY:f8cf0931cae6600f"} 12:34:04 [12:34:04.901] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:04 [12:34:04.934] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.935] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bb15aae1faad92e1 epochNumber=42 {"provingJobId":"1:BASE_PARITY:bb15aae1faad92e1"} 12:34:04 [12:34:04.936] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testqdie70/42 with map size 1073741824 12:34:04 [12:34:04.940] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:04 [12:34:04.972] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:06ddb7627725b907 epochNumber=1 {"provingJobId":"1:BASE_PARITY:06ddb7627725b907"} 12:34:04 [12:34:04.978] INFO: prover-client:proving-broker Proving Broker started 12:34:04 [12:34:04.979] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:723092fc8f25112c epochNumber=1 {"provingJobId":"1:BASE_PARITY:723092fc8f25112c"} 12:34:04 [12:34:04.981] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDkpL11/1 with map size 1073741824 12:34:04 [12:34:04.986] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.008] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c9660f1100e73df6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c9660f1100e73df6"} 12:34:05 [12:34:05.017] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.018] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d4bb411bf0044153 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d4bb411bf0044153"} 12:34:05 [12:34:05.019] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtJCQ3K/1 with map size 1073741824 12:34:05 [12:34:05.021] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.038] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d4bb411bf0044153 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:d4bb411bf0044153"} 12:34:05 [12:34:05.043] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.044] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2f891a3bfa55f6a3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2f891a3bfa55f6a3"} 12:34:05 [12:34:05.045] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testThplAL/1 with map size 1073741824 12:34:05 [12:34:05.048] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.070] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2f891a3bfa55f6a3 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:2f891a3bfa55f6a3"} 12:34:05 [12:34:05.071] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2f891a3bfa55f6a3 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:2f891a3bfa55f6a3"} 12:34:05 [12:34:05.080] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:2f891a3bfa55f6a3 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:2f891a3bfa55f6a3"} 12:34:05 [12:34:05.085] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.086] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:2815f38e1cc436f3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:2815f38e1cc436f3"} 12:34:05 [12:34:05.087] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYRuPKr/1 with map size 1073741824 12:34:05 [12:34:05.090] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.111] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.114] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.114] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:26d5186ed3762305 epochNumber=1 {"provingJobId":"1:BASE_PARITY:26d5186ed3762305"} 12:34:05 [12:34:05.115] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testp8TKQl/1 with map size 1073741824 12:34:05 [12:34:05.116] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.134] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:26d5186ed3762305 {"provingJobId":"1:BASE_PARITY:26d5186ed3762305"} 12:34:05 [12:34:05.134] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:26d5186ed3762305 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:26d5186ed3762305"} 12:34:05 [12:34:05.135] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:26d5186ed3762305 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:26d5186ed3762305"} 12:34:05 [12:34:05.144] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.145] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:883bc093e43c1dc6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:883bc093e43c1dc6"} 12:34:05 [12:34:05.145] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTfxgwK/1 with map size 1073741824 12:34:05 [12:34:05.147] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.165] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:883bc093e43c1dc6 {"provingJobId":"1:BASE_PARITY:883bc093e43c1dc6"} 12:34:05 [12:34:05.165] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:883bc093e43c1dc6 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:883bc093e43c1dc6"} 12:34:05 [12:34:05.175] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.176] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6855a83052bb0749 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6855a83052bb0749"} 12:34:05 [12:34:05.177] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testtv2AqN/1 with map size 1073741824 12:34:05 [12:34:05.180] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.197] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:6855a83052bb0749 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:6855a83052bb0749"} 12:34:05 [12:34:05.198] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6855a83052bb0749 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6855a83052bb0749"} 12:34:05 [12:34:05.207] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.208] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4eb95a6b622dffdf epochNumber=1 {"provingJobId":"1:BASE_PARITY:4eb95a6b622dffdf"} 12:34:05 [12:34:05.208] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnd9Woc/1 with map size 1073741824 12:34:05 [12:34:05.212] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.234] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4eb95a6b622dffdf type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4eb95a6b622dffdf"} 12:34:05 [12:34:05.234] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4eb95a6b622dffdf type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4eb95a6b622dffdf"} 12:34:05 [12:34:05.244] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.245] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1a3a5b7871549c25 epochNumber=0 {"provingJobId":"1:BASE_PARITY:1a3a5b7871549c25"} 12:34:05 [12:34:05.247] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testZ8uqAX/0 with map size 1073741824 12:34:05 [12:34:05.250] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.271] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1a3a5b7871549c25 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:1a3a5b7871549c25"} 12:34:05 [12:34:05.272] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:50612d9371e25c5f epochNumber=0 {"provingJobId":"1:BASE_PARITY:50612d9371e25c5f"} 12:34:05 [12:34:05.273] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1a3a5b7871549c25 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:1a3a5b7871549c25"} 12:34:05 [12:34:05.273] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0e91ab2d3bd39a61 epochNumber=0 {"provingJobId":"1:BASE_PARITY:0e91ab2d3bd39a61"} 12:34:05 [12:34:05.290] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.293] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.293] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ebe517317da669ad epochNumber=2 {"provingJobId":"1:BASE_PARITY:ebe517317da669ad"} 12:34:05 [12:34:05.294] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testl1H7hu/2 with map size 1073741824 12:34:05 [12:34:05.296] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.313] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:75ed7e1492ae3803 epochNumber=1 {"provingJobId":"1:BASE_PARITY:75ed7e1492ae3803"} 12:34:05 [12:34:05.314] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testl1H7hu/1 with map size 1073741824 12:34:05 [12:34:05.317] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.336] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.337] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:08fc1daeff701e35 epochNumber=1 {"provingJobId":"1:BASE_PARITY:08fc1daeff701e35"} 12:34:05 [12:34:05.338] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testP6ShRC/1 with map size 1073741824 12:34:05 [12:34:05.341] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.360] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.361] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8b5a857382f14db7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8b5a857382f14db7"} 12:34:05 [12:34:05.362] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUoGXp9/1 with map size 1073741824 12:34:05 [12:34:05.365] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.383] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b931526daf18ce00 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b931526daf18ce00"} 12:34:05 [12:34:05.388] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:63b5172d5aca2156 epochNumber=2 {"provingJobId":"1:BASE_PARITY:63b5172d5aca2156"} 12:34:05 [12:34:05.389] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testUoGXp9/2 with map size 1073741824 12:34:05 [12:34:05.391] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.411] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:ecf6b1fbb9e66d4e epochNumber=1 {"provingJobId":"1:BASE_PARITY:ecf6b1fbb9e66d4e"} 12:34:05 [12:34:05.425] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.426] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:067a6d33ce2dac39 epochNumber=1 {"provingJobId":"1:BASE_PARITY:067a6d33ce2dac39"} 12:34:05 [12:34:05.427] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testia2XqV/1 with map size 1073741824 12:34:05 [12:34:05.432] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.450] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c4228e3e38b918cb epochNumber=1 {"provingJobId":"1:BASE_PARITY:c4228e3e38b918cb"} 12:34:05 [12:34:05.458] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3277b220e767bce5 epochNumber=2 {"provingJobId":"1:BASE_PARITY:3277b220e767bce5"} 12:34:05 [12:34:05.459] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testia2XqV/2 with map size 1073741824 12:34:05 [12:34:05.465] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.489] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7b3b55372d78684c epochNumber=1 {"provingJobId":"1:BASE_PARITY:7b3b55372d78684c"} 12:34:05 [12:34:05.500] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.501] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dc999f293e6dbce1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dc999f293e6dbce1"} 12:34:05 [12:34:05.502] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMAqrDy/1 with map size 1073741824 12:34:05 [12:34:05.505] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.528] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:900f143e0e0b6c3b epochNumber=1 {"provingJobId":"1:BASE_PARITY:900f143e0e0b6c3b"} 12:34:05 [12:34:05.535] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5876f050b7b103ef epochNumber=2 {"provingJobId":"1:BASE_PARITY:5876f050b7b103ef"} 12:34:05 [12:34:05.536] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testMAqrDy/2 with map size 1073741824 12:34:05 [12:34:05.539] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.556] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:23de31ad5ce1a9b8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:23de31ad5ce1a9b8"} 12:34:05 [12:34:05.569] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.570] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aef057e1d1321472 epochNumber=1 {"provingJobId":"1:BASE_PARITY:aef057e1d1321472"} 12:34:05 [12:34:05.571] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWM5XHv/1 with map size 1073741824 12:34:05 [12:34:05.573] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.591] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0cd3a870d917f8e8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0cd3a870d917f8e8"} 12:34:05 [12:34:05.597] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:aef057e1d1321472 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:aef057e1d1321472"} 12:34:05 [12:34:05.608] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.610] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e8553b83a2391ca7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e8553b83a2391ca7"} 12:34:05 [12:34:05.612] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBJ4NRG/1 with map size 1073741824 12:34:05 [12:34:05.616] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.638] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:72505fad80c5ace5 epochNumber=1 {"provingJobId":"1:BASE_PARITY:72505fad80c5ace5"} 12:34:05 [12:34:05.644] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:e8553b83a2391ca7 type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:e8553b83a2391ca7"} 12:34:05 [12:34:05.655] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.656] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f06900a8cd7aac97 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f06900a8cd7aac97"} 12:34:05 [12:34:05.656] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5R7cFg/1 with map size 1073741824 12:34:05 [12:34:05.660] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.686] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1beb999cd63b4780 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1beb999cd63b4780"} 12:34:05 [12:34:05.694] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f06900a8cd7aac97 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:f06900a8cd7aac97"} 12:34:05 [12:34:05.698] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.699] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5483e57ba6ed9033 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5483e57ba6ed9033"} 12:34:05 [12:34:05.700] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXcJans/1 with map size 1073741824 12:34:05 [12:34:05.704] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.726] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5483e57ba6ed9033 {"provingJobId":"1:BASE_PARITY:5483e57ba6ed9033"} 12:34:05 [12:34:05.726] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5483e57ba6ed9033 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:5483e57ba6ed9033"} 12:34:05 [12:34:05.735] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a8964e371e02c757 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a8964e371e02c757"} 12:34:05 [12:34:05.742] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5483e57ba6ed9033 has already been completed {"provingJobId":"1:BASE_PARITY:5483e57ba6ed9033"} 12:34:05 [12:34:05.747] INFO: prover-client:proving-broker Proving Broker started 12:34:05 [12:34:05.748] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7fce81b668fe509b epochNumber=1 {"provingJobId":"1:BASE_PARITY:7fce81b668fe509b"} 12:34:05 [12:34:05.751] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testB1IOnk/1 with map size 1073741824 12:34:05 [12:34:05.754] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:05 [12:34:05.774] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d32bb51f31c71f02 epochNumber=2 {"provingJobId":"1:BASE_PARITY:d32bb51f31c71f02"} 12:34:05 [12:34:05.776] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testB1IOnk/2 with map size 1073741824 12:34:05 [12:34:05.778] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:06 [12:34:06.849] INFO: prover-client:proving-broker Proving Broker started 12:34:06 [12:34:06.855] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7fce81b668fe509b settled=false {"provingJobId":"1:BASE_PARITY:7fce81b668fe509b","status":"pending"} 12:34:06 [12:34:06.861] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d32bb51f31c71f02 settled=false {"provingJobId":"1:BASE_PARITY:d32bb51f31c71f02","status":"pending"} 12:34:06 [12:34:06.863] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:7fce81b668fe509b type=BASE_PARITY startedAt=1750163645798 older agent has taken job {"provingJobId":"1:BASE_PARITY:7fce81b668fe509b"} 12:34:06 [12:34:06.864] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:7fce81b668fe509b type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:7fce81b668fe509b"} 12:34:06 [12:34:06.872] INFO: prover-client:proving-broker Proving Broker started 12:34:06 [12:34:06.874] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:11acc71fc272f577 epochNumber=1 {"provingJobId":"1:BASE_PARITY:11acc71fc272f577"} 12:34:06 [12:34:06.878] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testR8ygTX/1 with map size 1073741824 12:34:06 [12:34:06.880] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:06 [12:34:06.899] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:36a843bbbe0dca04 epochNumber=2 {"provingJobId":"1:BASE_PARITY:36a843bbbe0dca04"} 12:34:06 [12:34:06.900] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testR8ygTX/2 with map size 1073741824 12:34:06 [12:34:06.902] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:07 [12:34:07.921] INFO: prover-client:proving-broker Proving Broker started 12:34:07 [12:34:07.925] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:11acc71fc272f577 settled=false {"provingJobId":"1:BASE_PARITY:11acc71fc272f577","status":"pending"} 12:34:07 [12:34:07.928] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:36a843bbbe0dca04 settled=false {"provingJobId":"1:BASE_PARITY:36a843bbbe0dca04","status":"pending"} 12:34:07 [12:34:07.929] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:11acc71fc272f577 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:11acc71fc272f577"} 12:34:07 [12:34:07.934] INFO: prover-client:proving-broker Proving Broker started 12:34:07 [12:34:07.934] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03bf7df3904e52b8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:03bf7df3904e52b8"} 12:34:07 [12:34:07.935] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkl1lLK/1 with map size 1073741824 12:34:07 [12:34:07.938] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:07 [12:34:07.954] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f1c19199713f415f epochNumber=2 {"provingJobId":"1:BASE_PARITY:f1c19199713f415f"} 12:34:07 [12:34:07.956] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testkl1lLK/2 with map size 1073741824 12:34:07 [12:34:07.963] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:08 [12:34:08.991] INFO: prover-client:proving-broker Proving Broker started 12:34:08 [12:34:08.993] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:03bf7df3904e52b8 settled=false {"provingJobId":"1:BASE_PARITY:03bf7df3904e52b8","status":"pending"} 12:34:08 [12:34:08.995] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f1c19199713f415f settled=false {"provingJobId":"1:BASE_PARITY:f1c19199713f415f","status":"pending"} 12:34:08 [12:34:08.996] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:03bf7df3904e52b8 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:03bf7df3904e52b8"} 12:34:08 [12:34:08.996] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:03bf7df3904e52b8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:03bf7df3904e52b8"} 12:34:09 [12:34:09.006] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.006] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1931dc60fb9695aa epochNumber=1 {"provingJobId":"1:BASE_PARITY:1931dc60fb9695aa"} 12:34:09 [12:34:09.008] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8CkXsc/1 with map size 1073741824 12:34:09 [12:34:09.011] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.034] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:15167c741b8616d1 epochNumber=2 {"provingJobId":"1:BASE_PARITY:15167c741b8616d1"} 12:34:09 [12:34:09.036] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test8CkXsc/2 with map size 1073741824 12:34:09 [12:34:09.043] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.073] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:1931dc60fb9695aa type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:1931dc60fb9695aa"} 12:34:09 [12:34:09.085] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:15167c741b8616d1 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:15167c741b8616d1"} 12:34:09 [12:34:09.094] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.095] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cb523c18df8da135 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cb523c18df8da135"} 12:34:09 [12:34:09.095] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7yhssU/1 with map size 1073741824 12:34:09 [12:34:09.098] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.119] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:feb548f0f453f476 epochNumber=2 {"provingJobId":"1:BASE_PARITY:feb548f0f453f476"} 12:34:09 [12:34:09.120] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test7yhssU/2 with map size 1073741824 12:34:09 [12:34:09.122] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.138] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cb523c18df8da135 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cb523c18df8da135"} 12:34:09 [12:34:09.138] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cb523c18df8da135 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cb523c18df8da135"} 12:34:09 [12:34:09.145] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:feb548f0f453f476 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:feb548f0f453f476"} 12:34:09 [12:34:09.156] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.161] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:90f003eb9540ce1f err=test error {"provingJoId":"1:BASE_PARITY:90f003eb9540ce1f"} 12:34:09 [12:34:09.171] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.179] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:c6b9c6c8b8a4cc7c not found {"provingJobId":"1:BASE_PARITY:c6b9c6c8b8a4cc7c"} 12:34:09 [12:34:09.184] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.185] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:eb36630ae9d348bc epochNumber=1 {"provingJobId":"1:BASE_PARITY:eb36630ae9d348bc"} 12:34:09 [12:34:09.186] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testA2fbil/1 with map size 1073741824 12:34:09 [12:34:09.190] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.213] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.214] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b4dc2db0ea142b0e epochNumber=1 {"provingJobId":"1:BASE_PARITY:b4dc2db0ea142b0e"} 12:34:09 [12:34:09.216] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIYWWAE/1 with map size 1073741824 12:34:09 [12:34:09.219] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.368] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b4dc2db0ea142b0e timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:b4dc2db0ea142b0e"} 12:34:09 [12:34:09.381] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.382] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9f94c4ef3185018d epochNumber=1 {"provingJobId":"1:BASE_PARITY:9f94c4ef3185018d"} 12:34:09 [12:34:09.383] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwX8Nao/1 with map size 1073741824 12:34:09 [12:34:09.387] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.514] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9f94c4ef3185018d timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:9f94c4ef3185018d"} 12:34:09 [12:34:09.515] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1be54f6b8dfa1f31 epochNumber=10 {"provingJobId":"1:BASE_PARITY:1be54f6b8dfa1f31"} 12:34:09 [12:34:09.517] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testwX8Nao/10 with map size 1073741824 12:34:09 [12:34:09.521] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.566] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:09 [12:34:09.568] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 12:34:09 [12:34:09.677] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.678] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e04fef4987a3de20 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e04fef4987a3de20"} 12:34:09 [12:34:09.679] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfuVycm/1 with map size 1073741824 12:34:09 [12:34:09.684] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.856] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e04fef4987a3de20 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:e04fef4987a3de20"} 12:34:09 [12:34:09.906] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.907] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c82bec3b0a68553c epochNumber=1 {"provingJobId":"1:BASE_PARITY:c82bec3b0a68553c"} 12:34:09 [12:34:09.907] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUQDoLd/1 with map size 1073741824 12:34:09 [12:34:09.909] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.929] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:c82bec3b0a68553c type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:c82bec3b0a68553c"} 12:34:09 [12:34:09.936] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.937] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3445b2b24558c5ff epochNumber=1 {"provingJobId":"1:BASE_PARITY:3445b2b24558c5ff"} 12:34:09 [12:34:09.938] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testugjcZb/1 with map size 1073741824 12:34:09 [12:34:09.942] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:09 [12:34:09.975] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:3445b2b24558c5ff type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:3445b2b24558c5ff"} 12:34:09 [12:34:09.976] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3445b2b24558c5ff type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:3445b2b24558c5ff"} 12:34:09 [12:34:09.985] INFO: prover-client:proving-broker Proving Broker started 12:34:09 [12:34:09.986] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c2131610867776c4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c2131610867776c4"} 12:34:09 [12:34:09.987] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyoolfW/1 with map size 1073741824 12:34:09 [12:34:09.988] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.006] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c2131610867776c4 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:c2131610867776c4"} 12:34:10 [12:34:10.019] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.020] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bcf206688999c214 epochNumber=1 {"provingJobId":"1:BASE_PARITY:bcf206688999c214"} 12:34:10 [12:34:10.021] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjEczVr/1 with map size 1073741824 12:34:10 [12:34:10.027] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.049] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:bcf206688999c214 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:bcf206688999c214"} 12:34:10 [12:34:10.050] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7fcfff9c82f8c165 epochNumber=3 {"provingJobId":"1:BASE_PARITY:7fcfff9c82f8c165"} 12:34:10 [12:34:10.052] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testjEczVr/3 with map size 1073741824 12:34:10 [12:34:10.054] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.095] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:10 [12:34:10.095] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 12:34:10 [12:34:10.112] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:bcf206688999c214 err=test error {"provingJoId":"1:BASE_PARITY:bcf206688999c214"} 12:34:10 [12:34:10.118] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testLCymkU/1 with map size 1073741824 12:34:10 [12:34:10.124] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.154] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testLCymkU/2 with map size 1073741824 12:34:10 [12:34:10.160] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.188] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.193] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cb976664adddaa6a settled=false {"provingJobId":"1:BASE_PARITY:cb976664adddaa6a","status":"pending"} 12:34:10 [12:34:10.195] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:368c4bb4bb4c1978 settled=false {"provingJobId":"1:BASE_PARITY:368c4bb4bb4c1978","status":"pending"} 12:34:10 [12:34:10.204] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWPwD2o/1 with map size 1073741824 12:34:10 [12:34:10.206] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.228] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWPwD2o/2 with map size 1073741824 12:34:10 [12:34:10.232] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.268] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.272] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:15a1813e9f78342b settled=true {"provingJobId":"1:BASE_PARITY:15a1813e9f78342b","status":"fulfilled"} 12:34:10 [12:34:10.276] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:c5b80968097e9552 settled=true {"provingJobId":"2:BASE_PARITY:c5b80968097e9552","status":"fulfilled"} 12:34:10 [12:34:10.283] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNGzyex/1 with map size 1073741824 12:34:10 [12:34:10.286] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.312] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNGzyex/2 with map size 1073741824 12:34:10 [12:34:10.315] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.334] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.336] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:031b3cb6eba42b9e settled=true {"provingJobId":"1:BASE_PARITY:031b3cb6eba42b9e","status":"fulfilled"} 12:34:10 [12:34:10.338] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e70c1fed485e3aec settled=false {"provingJobId":"1:BASE_PARITY:e70c1fed485e3aec","status":"pending"} 12:34:10 [12:34:10.343] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.344] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:59ffb9124193f402 epochNumber=1 {"provingJobId":"1:BASE_PARITY:59ffb9124193f402"} 12:34:10 [12:34:10.344] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWDnmdC/1 with map size 1073741824 12:34:10 [12:34:10.347] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.377] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.378] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:de4617d34859ddb4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:de4617d34859ddb4"} 12:34:10 [12:34:10.378] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:de4617d34859ddb4: Error: db error: Error: db error 12:34:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:de4617d34859ddb4"} 12:34:10 [12:34:10.386] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.387] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b28d8de76b6d6152 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b28d8de76b6d6152"} 12:34:10 [12:34:10.388] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testXXJQau/1 with map size 1073741824 12:34:10 [12:34:10.391] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.412] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b28d8de76b6d6152 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b28d8de76b6d6152"} 12:34:10 [12:34:10.413] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b28d8de76b6d6152 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:b28d8de76b6d6152"} 12:34:10 [12:34:10.426] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.426] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5ea25a71afd54ae7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:5ea25a71afd54ae7"} 12:34:10 [12:34:10.427] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testx7Z0LS/1 with map size 1073741824 12:34:10 [12:34:10.430] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.454] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:5ea25a71afd54ae7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:5ea25a71afd54ae7"} 12:34:10 [12:34:10.454] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5ea25a71afd54ae7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:5ea25a71afd54ae7"} 12:34:10 [12:34:10.456] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:5ea25a71afd54ae7: Error: db error 12:34:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:5ea25a71afd54ae7"} 12:34:10 [12:34:10.460] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.461] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:04b99abcac9f2918 epochNumber=1 {"provingJobId":"1:BASE_PARITY:04b99abcac9f2918"} 12:34:10 [12:34:10.463] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testugQj5Z/1 with map size 1073741824 12:34:10 [12:34:10.466] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.487] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:04b99abcac9f2918 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:04b99abcac9f2918"} 12:34:10 [12:34:10.488] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:04b99abcac9f2918 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:04b99abcac9f2918"} 12:34:10 [12:34:10.500] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.501] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0de89e2bf1478205 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0de89e2bf1478205"} 12:34:10 [12:34:10.502] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testGGcxUi/1 with map size 1073741824 12:34:10 [12:34:10.504] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.526] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0de89e2bf1478205 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0de89e2bf1478205"} 12:34:10 [12:34:10.527] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0de89e2bf1478205 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:0de89e2bf1478205"} 12:34:10 [12:34:10.528] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:0de89e2bf1478205 jobErr=test error: Error: db error 12:34:10 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:0de89e2bf1478205"} 12:34:10 [12:34:10.533] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.533] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1a0acaded14ed855 not found {"provingJobId":"1:BASE_PARITY:1a0acaded14ed855"} 12:34:10 [12:34:10.538] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.538] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:dec5bdcb68ba78d4 err=test error {"provingJoId":"1:BASE_PARITY:dec5bdcb68ba78d4"} 12:34:10 [12:34:10.545] INFO: prover-client:proving-broker Proving Broker started 12:34:10 [12:34:10.556] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0394c433fad01e4b epochNumber=1 {"provingJobId":"1:BASE_PARITY:0394c433fad01e4b"} 12:34:10 [12:34:10.557] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOkToh6/1 with map size 1073741824 12:34:10 [12:34:10.559] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.593] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0394c433fad01e4b type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0394c433fad01e4b"} 12:34:10 [12:34:10.594] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0394c433fad01e4b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0394c433fad01e4b"} 12:34:10 [12:34:10.611] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:9e35210e682fc2f8 epochNumber=2 {"provingJobId":"2:BASE_PARITY:9e35210e682fc2f8"} 12:34:10 [12:34:10.613] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testOkToh6/2 with map size 1073741824 12:34:10 [12:34:10.619] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.642] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:9e35210e682fc2f8 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:9e35210e682fc2f8"} 12:34:10 [12:34:10.643] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:9e35210e682fc2f8 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:9e35210e682fc2f8"} 12:34:10 [12:34:10.714] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:91a307fad0018d83 epochNumber=3 {"provingJobId":"3:BASE_PARITY:91a307fad0018d83"} 12:34:10 [12:34:10.716] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testOkToh6/3 with map size 1073741824 12:34:10 [12:34:10.721] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.748] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:10 [12:34:10.749] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 12:34:10 [12:34:10.806] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:78482b174e2d8d2f epochNumber=4 {"provingJobId":"4:BASE_PARITY:78482b174e2d8d2f"} 12:34:10 [12:34:10.808] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testOkToh6/4 with map size 1073741824 12:34:10 [12:34:10.812] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.854] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:10 [12:34:10.855] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 12:34:10 [12:34:10.898] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:f30eeb68b3b73fc4 epochNumber=5 {"provingJobId":"5:BASE_PARITY:f30eeb68b3b73fc4"} 12:34:10 [12:34:10.900] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testOkToh6/5 with map size 1073741824 12:34:10 [12:34:10.904] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 12:34:10 [12:34:10.940] INFO: prover-client:proving-broker Cleaned up jobs=1 12:34:10 [12:34:10.941] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 12:34:10 PASS src/proving_broker/proving_broker.test.ts (13.719 s) 12:34:10 ProvingBroker 12:34:10 Producer API 12:34:10 ✓ refuses stale jobs (59 ms) 12:34:10 ✓ enqueues jobs (5 ms) 12:34:10 ✓ ignores duplicate jobs (4 ms) 12:34:11 ✓ reports correct status when enqueuing repeat jobs (5 ms) 12:34:11 ✓ throws an error in case of duplicate job IDs (36 ms) 12:34:11 ✓ returns not-found status for non-existing jobs (2 ms) 12:34:11 ✓ cancels jobs in queue (3 ms) 12:34:11 ✓ cancels jobs in-progress (2 ms) 12:34:11 ✓ returns job result if successful (14 ms) 12:34:11 ✓ returns job error if failed (8 ms) 12:34:11 ✓ correctly returns job status for concurrent writes (3 ms) 12:34:11 ✓ refuses stale jobs (44 ms) 12:34:11 ✓ enqueues jobs (40 ms) 12:34:11 ✓ ignores duplicate jobs (25 ms) 12:34:11 ✓ reports correct status when enqueuing repeat jobs (40 ms) 12:34:11 ✓ throws an error in case of duplicate job IDs (28 ms) 12:34:11 ✓ returns not-found status for non-existing jobs (2 ms) 12:34:11 ✓ cancels jobs in queue (29 ms) 12:34:11 ✓ cancels jobs in-progress (30 ms) 12:34:11 ✓ returns job result if successful (31 ms) 12:34:11 ✓ returns job error if failed (37 ms) 12:34:11 ✓ correctly returns job status for concurrent writes (46 ms) 12:34:11 Consumer API 12:34:11 ✓ returns undefined if no jobs are available (1 ms) 12:34:11 ✓ returns jobs in priority order (3 ms) 12:34:11 ✓ returns undefined if no jobs are available for the given allowList (2 ms) 12:34:11 ✓ returns a job if it is in the allowList (2 ms) 12:34:11 ✓ returns the most important job if it is in the allowList (2 ms) 12:34:11 ✓ returns any job if filter is empty (3 ms) 12:34:11 ✓ returns a new job when reporting job success (3 ms) 12:34:11 ✓ returns a new job when reporting permanent error (2 ms) 12:34:11 ✓ returns a new job when reporting retry-able error (2 ms) 12:34:11 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 12:34:11 ✓ returns a new job if job is already in progress elsewhere (1057 ms) 12:34:11 ✓ avoids sending the same job to a new agent after a restart (1004 ms) 12:34:11 ✓ avoids sending a completed job to a new agent after a restart (1004 ms) 12:34:11 ✓ tracks job result if in progress (2 ms) 12:34:11 ✓ tracks job result even if job is in queue (1 ms) 12:34:11 ✓ ignores reported job error if unknown job (1 ms) 12:34:11 ✓ ignores job result if unknown job (1 ms) 12:34:11 ✓ returns undefined if no jobs are available (3 ms) 12:34:11 ✓ returns jobs in priority order (44 ms) 12:34:11 ✓ returns undefined if no jobs are available for the given allowList (23 ms) 12:34:11 ✓ returns a job if it is in the allowList (61 ms) 12:34:11 ✓ returns the most important job if it is in the allowList (77 ms) 12:34:11 ✓ returns any job if filter is empty (68 ms) 12:34:11 ✓ returns a new job when reporting job success (38 ms) 12:34:11 ✓ returns a new job when reporting permanent error (46 ms) 12:34:11 ✓ returns a new job when reporting retry-able error (43 ms) 12:34:11 ✓ returns a new job when reporting progress if current one is cancelled (49 ms) 12:34:11 ✓ returns a new job if job is already in progress elsewhere (1122 ms) 12:34:11 ✓ avoids sending the same job to a new agent after a restart (1063 ms) 12:34:11 ✓ avoids sending a completed job to a new agent after a restart (1070 ms) 12:34:11 ✓ tracks job result if in progress (90 ms) 12:34:11 ✓ tracks job result even if job is in queue (62 ms) 12:34:11 ✓ ignores reported job error if unknown job (11 ms) 12:34:11 ✓ ignores job result if unknown job (14 ms) 12:34:11 Timeouts 12:34:11 ✓ tracks in progress jobs (1 ms) 12:34:11 ✓ re-enqueues jobs that time out (109 ms) 12:34:11 ✓ cancel stale jobs that time out (230 ms) 12:34:11 ✓ keeps the jobs in progress while it is alive (203 ms) 12:34:11 ✓ tracks in progress jobs (30 ms) 12:34:11 ✓ re-enqueues jobs that time out (165 ms) 12:34:11 ✓ cancel stale jobs that time out (298 ms) 12:34:11 ✓ keeps the jobs in progress while it is alive (229 ms) 12:34:11 Retries 12:34:11 ✓ retries jobs (3 ms) 12:34:11 ✓ retries up to a maximum number of times (3 ms) 12:34:11 ✓ passing retry=false does not retry (2 ms) 12:34:11 ✓ does not retry if job is stale (30 ms) 12:34:11 ✓ retries jobs (26 ms) 12:34:11 ✓ retries up to a maximum number of times (52 ms) 12:34:11 ✓ passing retry=false does not retry (33 ms) 12:34:11 ✓ does not retry if job is stale (97 ms) 12:34:11 Database management 12:34:11 ✓ re-enqueues proof requests on start (3 ms) 12:34:11 ✓ restores proof results on start (2 ms) 12:34:11 ✓ only re-enqueues unfinished jobs (2 ms) 12:34:11 ✓ saves job when enqueued (2 ms) 12:34:11 ✓ does not retain job if database fails to save (2 ms) 12:34:11 ✓ saves job result (1 ms) 12:34:11 ✓ saves result even if database fails to save (2 ms) 12:34:11 ✓ saves job error (2 ms) 12:34:11 ✓ saves job error even if database fails to save (2 ms) 12:34:11 ✓ does not save job result if job is unknown (3 ms) 12:34:11 ✓ does not save job error if job is unknown (2 ms) 12:34:11 ✓ cleans up old jobs periodically (262 ms) 12:34:11 ✓ re-enqueues proof requests on start (85 ms) 12:34:11 ✓ restores proof results on start (79 ms) 12:34:11 ✓ only re-enqueues unfinished jobs (60 ms) 12:34:11 ✓ saves job when enqueued (33 ms) 12:34:11 ✓ does not retain job if database fails to save (7 ms) 12:34:11 ✓ saves job result (41 ms) 12:34:11 ✓ saves result even if database fails to save (35 ms) 12:34:11 ✓ saves job error (39 ms) 12:34:11 ✓ saves job error even if database fails to save (32 ms) 12:34:11 ✓ does not save job result if job is unknown (5 ms) 12:34:11 ✓ does not save job error if job is unknown (5 ms) 12:34:11 ✓ cleans up old jobs periodically (440 ms) 12:34:11 12:34:11 Test Suites: 1 passed, 1 total 12:34:11 Tests: 96 passed, 96 total 12:34:11 Snapshots: 0 total 12:34:11 Time: 13.833 s 12:34:11 Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts. 12:34:11 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?