ACI ·

Parent Log: http://ci.aztec-labs.com/ad41d2c01b79593b Command: 21081b81c02e2d2a yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/c06f30929f18b85bcea93c0926a8d640a3819f00 Env: REF_NAME=gh-readonly-queue/master/pr-15074-2707ec99de02004f3e30e7af0810e0d741ab9794 CURRENT_VERSION=0.87.8 CI_FULL=1 Date: Mon Jun 16 16:21:34 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15074_amd64_x2-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b95296ef06707e5f_master 16:21:38 [16:21:38.033] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.041] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6236d5a6f1f72239 epochNumber=42 {"provingJobId":"1:BASE_PARITY:6236d5a6f1f72239"} 16:21:38 [16:21:38.044] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:c5eb0670baf066b3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c5eb0670baf066b3"} 16:21:38 [16:21:38.067] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.068] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:668180d2fb05b2eb epochNumber=1 {"provingJobId":"1:BASE_PARITY:668180d2fb05b2eb"} 16:21:38 [16:21:38.069] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:273a632bf9775b83 epochNumber=1 {"provingJobId":"1:BASE_PARITY:273a632bf9775b83"} 16:21:38 [16:21:38.070] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.070] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f679a5334a9ef742 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f679a5334a9ef742"} 16:21:38 [16:21:38.071] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f679a5334a9ef742 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f679a5334a9ef742"} 16:21:38 [16:21:38.073] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.073] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6400c40757c9be9f epochNumber=1 {"provingJobId":"1:BASE_PARITY:6400c40757c9be9f"} 16:21:38 [16:21:38.074] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6400c40757c9be9f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:6400c40757c9be9f"} 16:21:38 [16:21:38.075] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:6400c40757c9be9f type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:6400c40757c9be9f"} 16:21:38 [16:21:38.075] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:6400c40757c9be9f epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:6400c40757c9be9f"} 16:21:38 [16:21:38.076] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.077] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a7c7d04105abcec0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a7c7d04105abcec0"} 16:21:38 [16:21:38.095] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.096] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.097] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1056cdbd169f7e61 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1056cdbd169f7e61"} 16:21:38 [16:21:38.098] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:1056cdbd169f7e61 {"provingJobId":"1:BASE_PARITY:1056cdbd169f7e61"} 16:21:38 [16:21:38.098] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:1056cdbd169f7e61 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:1056cdbd169f7e61"} 16:21:38 [16:21:38.098] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1056cdbd169f7e61 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:1056cdbd169f7e61"} 16:21:38 [16:21:38.100] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.100] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d10c9700081dd827 epochNumber=1 {"provingJobId":"1:BASE_PARITY:d10c9700081dd827"} 16:21:38 [16:21:38.101] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:d10c9700081dd827 {"provingJobId":"1:BASE_PARITY:d10c9700081dd827"} 16:21:38 [16:21:38.101] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d10c9700081dd827 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:d10c9700081dd827"} 16:21:38 [16:21:38.102] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.103] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:80192ddba98d8c60 epochNumber=1 {"provingJobId":"1:BASE_PARITY:80192ddba98d8c60"} 16:21:38 [16:21:38.104] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:80192ddba98d8c60 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:80192ddba98d8c60"} 16:21:38 [16:21:38.104] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:80192ddba98d8c60 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:80192ddba98d8c60"} 16:21:38 [16:21:38.105] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.105] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:42580e7d35e38a05 epochNumber=1 {"provingJobId":"1:BASE_PARITY:42580e7d35e38a05"} 16:21:38 [16:21:38.106] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:42580e7d35e38a05 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:42580e7d35e38a05"} 16:21:38 [16:21:38.106] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:42580e7d35e38a05 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:42580e7d35e38a05"} 16:21:38 [16:21:38.107] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.107] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bd8c00dee2029d04 epochNumber=0 {"provingJobId":"1:BASE_PARITY:bd8c00dee2029d04"} 16:21:38 [16:21:38.108] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:bd8c00dee2029d04 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:bd8c00dee2029d04"} 16:21:38 [16:21:38.108] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:376b5d715e945ff0 epochNumber=0 {"provingJobId":"1:BASE_PARITY:376b5d715e945ff0"} 16:21:38 [16:21:38.108] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:bd8c00dee2029d04 epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:bd8c00dee2029d04"} 16:21:38 [16:21:38.108] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:680f9cfbc399a0ae epochNumber=0 {"provingJobId":"1:BASE_PARITY:680f9cfbc399a0ae"} 16:21:38 [16:21:38.109] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.110] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.111] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:21c4ff47587eb432 epochNumber=2 {"provingJobId":"1:BASE_PARITY:21c4ff47587eb432"} 16:21:38 [16:21:38.111] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:30bb69f3fdcd8972 epochNumber=1 {"provingJobId":"1:BASE_PARITY:30bb69f3fdcd8972"} 16:21:38 [16:21:38.112] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.130] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a6560ab5def5bc8e epochNumber=1 {"provingJobId":"1:BASE_PARITY:a6560ab5def5bc8e"} 16:21:38 [16:21:38.132] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.132] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:50870ec79cf70e50 epochNumber=1 {"provingJobId":"1:BASE_PARITY:50870ec79cf70e50"} 16:21:38 [16:21:38.133] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7395a39e5795b6d8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7395a39e5795b6d8"} 16:21:38 [16:21:38.133] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:90d5a2057fb15217 epochNumber=2 {"provingJobId":"1:BASE_PARITY:90d5a2057fb15217"} 16:21:38 [16:21:38.133] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f0f9bf8b994af0fd epochNumber=1 {"provingJobId":"1:BASE_PARITY:f0f9bf8b994af0fd"} 16:21:38 [16:21:38.134] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.135] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1095a4c91ec344cc epochNumber=1 {"provingJobId":"1:BASE_PARITY:1095a4c91ec344cc"} 16:21:38 [16:21:38.136] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:686fc22202be6016 epochNumber=1 {"provingJobId":"1:BASE_PARITY:686fc22202be6016"} 16:21:38 [16:21:38.136] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:490b2ac2c25ae92f epochNumber=2 {"provingJobId":"1:BASE_PARITY:490b2ac2c25ae92f"} 16:21:38 [16:21:38.136] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3cafe939ecccd6ed epochNumber=1 {"provingJobId":"1:BASE_PARITY:3cafe939ecccd6ed"} 16:21:38 [16:21:38.138] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.138] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9d46e7e83697e232 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9d46e7e83697e232"} 16:21:38 [16:21:38.138] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:771a5595d254cbbd epochNumber=1 {"provingJobId":"1:BASE_PARITY:771a5595d254cbbd"} 16:21:38 [16:21:38.139] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:78f4abdf5611145e epochNumber=2 {"provingJobId":"1:BASE_PARITY:78f4abdf5611145e"} 16:21:38 [16:21:38.139] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:131ec371f19eb5a0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:131ec371f19eb5a0"} 16:21:38 [16:21:38.140] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.140] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7d00cad4701a95e6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7d00cad4701a95e6"} 16:21:38 [16:21:38.141] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c0e6466cf0c9995e epochNumber=1 {"provingJobId":"1:BASE_PARITY:c0e6466cf0c9995e"} 16:21:38 [16:21:38.141] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:7d00cad4701a95e6 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:7d00cad4701a95e6"} 16:21:38 [16:21:38.142] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.143] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8237be028bc273ff epochNumber=1 {"provingJobId":"1:BASE_PARITY:8237be028bc273ff"} 16:21:38 [16:21:38.143] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03411be529ec57f8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:03411be529ec57f8"} 16:21:38 [16:21:38.143] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8237be028bc273ff type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:8237be028bc273ff"} 16:21:38 [16:21:38.144] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.144] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fa0dcf77bc29b8f1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:fa0dcf77bc29b8f1"} 16:21:38 [16:21:38.145] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dc9123248bf3375e epochNumber=1 {"provingJobId":"1:BASE_PARITY:dc9123248bf3375e"} 16:21:38 [16:21:38.145] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fa0dcf77bc29b8f1 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:fa0dcf77bc29b8f1"} 16:21:38 [16:21:38.146] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.146] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0c71e11f97c318b9 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0c71e11f97c318b9"} 16:21:38 [16:21:38.146] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:0c71e11f97c318b9 {"provingJobId":"1:BASE_PARITY:0c71e11f97c318b9"} 16:21:38 [16:21:38.147] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0c71e11f97c318b9 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:0c71e11f97c318b9"} 16:21:38 [16:21:38.147] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:00021b7d3109aa0c epochNumber=1 {"provingJobId":"1:BASE_PARITY:00021b7d3109aa0c"} 16:21:38 [16:21:38.147] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0c71e11f97c318b9 has already been completed {"provingJobId":"1:BASE_PARITY:0c71e11f97c318b9"} 16:21:38 [16:21:38.148] INFO: prover-client:proving-broker Proving Broker started 16:21:38 [16:21:38.148] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e9c9b919d5ce50f6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e9c9b919d5ce50f6"} 16:21:38 [16:21:38.149] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9a57a9e0b92f7cee epochNumber=2 {"provingJobId":"1:BASE_PARITY:9a57a9e0b92f7cee"} 16:21:39 [16:21:39.201] INFO: prover-client:proving-broker Proving Broker started 16:21:39 [16:21:39.201] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e9c9b919d5ce50f6 settled=false {"provingJobId":"1:BASE_PARITY:e9c9b919d5ce50f6","status":"pending"} 16:21:39 [16:21:39.202] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9a57a9e0b92f7cee settled=false {"provingJobId":"1:BASE_PARITY:9a57a9e0b92f7cee","status":"pending"} 16:21:39 [16:21:39.203] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:e9c9b919d5ce50f6 type=BASE_PARITY startedAt=1750090898149 older agent has taken job {"provingJobId":"1:BASE_PARITY:e9c9b919d5ce50f6"} 16:21:39 [16:21:39.203] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e9c9b919d5ce50f6 type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:e9c9b919d5ce50f6"} 16:21:39 [16:21:39.205] INFO: prover-client:proving-broker Proving Broker started 16:21:39 [16:21:39.205] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e8234d52dd2c16a6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:e8234d52dd2c16a6"} 16:21:39 [16:21:39.206] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6a45e8be0103e5fe epochNumber=2 {"provingJobId":"1:BASE_PARITY:6a45e8be0103e5fe"} 16:21:40 [16:21:40.207] INFO: prover-client:proving-broker Proving Broker started 16:21:40 [16:21:40.207] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e8234d52dd2c16a6 settled=false {"provingJobId":"1:BASE_PARITY:e8234d52dd2c16a6","status":"pending"} 16:21:40 [16:21:40.208] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6a45e8be0103e5fe settled=false {"provingJobId":"1:BASE_PARITY:6a45e8be0103e5fe","status":"pending"} 16:21:40 [16:21:40.208] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e8234d52dd2c16a6 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:e8234d52dd2c16a6"} 16:21:40 [16:21:40.210] INFO: prover-client:proving-broker Proving Broker started 16:21:40 [16:21:40.211] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d33b34054c95652d epochNumber=1 {"provingJobId":"1:BASE_PARITY:d33b34054c95652d"} 16:21:40 [16:21:40.211] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e8b4f052b829300b epochNumber=2 {"provingJobId":"1:BASE_PARITY:e8b4f052b829300b"} 16:21:41 [16:21:41.214] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.216] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d33b34054c95652d settled=false {"provingJobId":"1:BASE_PARITY:d33b34054c95652d","status":"pending"} 16:21:41 [16:21:41.216] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e8b4f052b829300b settled=false {"provingJobId":"1:BASE_PARITY:e8b4f052b829300b","status":"pending"} 16:21:41 [16:21:41.218] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d33b34054c95652d type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d33b34054c95652d"} 16:21:41 [16:21:41.219] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d33b34054c95652d type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d33b34054c95652d"} 16:21:41 [16:21:41.222] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.223] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:53b09480d669d3e2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:53b09480d669d3e2"} 16:21:41 [16:21:41.224] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4877259ccd081abe epochNumber=2 {"provingJobId":"1:BASE_PARITY:4877259ccd081abe"} 16:21:41 [16:21:41.225] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:53b09480d669d3e2 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:53b09480d669d3e2"} 16:21:41 [16:21:41.230] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4877259ccd081abe type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4877259ccd081abe"} 16:21:41 [16:21:41.232] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.233] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:59b5553b5e2a64bf epochNumber=1 {"provingJobId":"1:BASE_PARITY:59b5553b5e2a64bf"} 16:21:41 [16:21:41.233] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8bd315f0a5debd35 epochNumber=2 {"provingJobId":"1:BASE_PARITY:8bd315f0a5debd35"} 16:21:41 [16:21:41.234] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:59b5553b5e2a64bf type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:59b5553b5e2a64bf"} 16:21:41 [16:21:41.234] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:59b5553b5e2a64bf type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:59b5553b5e2a64bf"} 16:21:41 [16:21:41.234] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8bd315f0a5debd35 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:8bd315f0a5debd35"} 16:21:41 [16:21:41.236] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.237] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:08e9c20d6d73375f err=test error {"provingJoId":"1:BASE_PARITY:08e9c20d6d73375f"} 16:21:41 [16:21:41.239] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.239] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:9d7f1f844edc940f not found {"provingJobId":"1:BASE_PARITY:9d7f1f844edc940f"} 16:21:41 [16:21:41.241] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.242] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:70f0f52649c16280 epochNumber=1 {"provingJobId":"1:BASE_PARITY:70f0f52649c16280"} 16:21:41 [16:21:41.244] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.244] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:44533335aa1fd911 epochNumber=1 {"provingJobId":"1:BASE_PARITY:44533335aa1fd911"} 16:21:41 [16:21:41.346] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:44533335aa1fd911 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:44533335aa1fd911"} 16:21:41 [16:21:41.353] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.354] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:715f081062d17b64 epochNumber=1 {"provingJobId":"1:BASE_PARITY:715f081062d17b64"} 16:21:41 [16:21:41.479] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:715f081062d17b64 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:715f081062d17b64"} 16:21:41 [16:21:41.480] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f60e44992340392d epochNumber=10 {"provingJobId":"1:BASE_PARITY:f60e44992340392d"} 16:21:41 [16:21:41.505] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:41 [16:21:41.608] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.609] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:57cfcd1f8f6a276f epochNumber=1 {"provingJobId":"1:BASE_PARITY:57cfcd1f8f6a276f"} 16:21:41 [16:21:41.786] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:57cfcd1f8f6a276f timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:57cfcd1f8f6a276f"} 16:21:41 [16:21:41.814] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.815] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9c21ce68edff69dd epochNumber=1 {"provingJobId":"1:BASE_PARITY:9c21ce68edff69dd"} 16:21:41 [16:21:41.816] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:9c21ce68edff69dd type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:9c21ce68edff69dd"} 16:21:41 [16:21:41.818] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.818] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7e024b134160fba4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:7e024b134160fba4"} 16:21:41 [16:21:41.819] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:7e024b134160fba4 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:7e024b134160fba4"} 16:21:41 [16:21:41.820] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7e024b134160fba4 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:7e024b134160fba4"} 16:21:41 [16:21:41.822] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.822] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f961a8e3ee1829c4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f961a8e3ee1829c4"} 16:21:41 [16:21:41.823] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f961a8e3ee1829c4 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:f961a8e3ee1829c4"} 16:21:41 [16:21:41.823] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.824] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:129cc08823f7a984 epochNumber=1 {"provingJobId":"1:BASE_PARITY:129cc08823f7a984"} 16:21:41 [16:21:41.824] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:129cc08823f7a984 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:129cc08823f7a984"} 16:21:41 [16:21:41.825] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a4943d7bb02e0077 epochNumber=3 {"provingJobId":"1:BASE_PARITY:a4943d7bb02e0077"} 16:21:41 [16:21:41.848] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:41 [16:21:41.850] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:129cc08823f7a984 err=test error {"provingJoId":"1:BASE_PARITY:129cc08823f7a984"} 16:21:41 [16:21:41.852] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.852] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:8a9de84035134c3c settled=false {"provingJobId":"1:BASE_PARITY:8a9de84035134c3c","status":"pending"} 16:21:41 [16:21:41.853] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9326eae30e1dc7e3 settled=false {"provingJobId":"1:BASE_PARITY:9326eae30e1dc7e3","status":"pending"} 16:21:41 [16:21:41.855] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.856] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:acfa63df002f3790 settled=true {"provingJobId":"1:BASE_PARITY:acfa63df002f3790","status":"fulfilled"} 16:21:41 [16:21:41.856] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:184ab78934c3de7f settled=true {"provingJobId":"2:BASE_PARITY:184ab78934c3de7f","status":"fulfilled"} 16:21:41 [16:21:41.857] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.857] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0d7f8c7a60d7958f settled=true {"provingJobId":"1:BASE_PARITY:0d7f8c7a60d7958f","status":"fulfilled"} 16:21:41 [16:21:41.858] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a0ef63deaa2ed5ee settled=false {"provingJobId":"1:BASE_PARITY:a0ef63deaa2ed5ee","status":"pending"} 16:21:41 [16:21:41.859] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.861] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9e25a3be3809f275 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9e25a3be3809f275"} 16:21:41 [16:21:41.863] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.863] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:88519ba5fe994b9d epochNumber=1 {"provingJobId":"1:BASE_PARITY:88519ba5fe994b9d"} 16:21:41 [16:21:41.864] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:88519ba5fe994b9d: Error: db error: Error: db error 16:21:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:88519ba5fe994b9d"} 16:21:41 [16:21:41.865] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.865] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aa580252fdd2acb1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:aa580252fdd2acb1"} 16:21:41 [16:21:41.866] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:aa580252fdd2acb1 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:aa580252fdd2acb1"} 16:21:41 [16:21:41.866] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:aa580252fdd2acb1 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:aa580252fdd2acb1"} 16:21:41 [16:21:41.867] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.868] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:58fa445e84c73279 epochNumber=1 {"provingJobId":"1:BASE_PARITY:58fa445e84c73279"} 16:21:41 [16:21:41.868] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:58fa445e84c73279 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:58fa445e84c73279"} 16:21:41 [16:21:41.868] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:58fa445e84c73279 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:58fa445e84c73279"} 16:21:41 [16:21:41.869] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:58fa445e84c73279: Error: db error 16:21:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:58fa445e84c73279"} 16:21:41 [16:21:41.870] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.870] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:19c2ac87fcb3187f epochNumber=1 {"provingJobId":"1:BASE_PARITY:19c2ac87fcb3187f"} 16:21:41 [16:21:41.870] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:19c2ac87fcb3187f type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:19c2ac87fcb3187f"} 16:21:41 [16:21:41.870] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:19c2ac87fcb3187f type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:19c2ac87fcb3187f"} 16:21:41 [16:21:41.871] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.872] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b5f2d650986b9b58 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b5f2d650986b9b58"} 16:21:41 [16:21:41.872] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b5f2d650986b9b58 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b5f2d650986b9b58"} 16:21:41 [16:21:41.872] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b5f2d650986b9b58 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b5f2d650986b9b58"} 16:21:41 [16:21:41.872] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:b5f2d650986b9b58 jobErr=test error: Error: db error 16:21:41 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:b5f2d650986b9b58"} 16:21:41 [16:21:41.873] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.874] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d2ec268659721488 not found {"provingJobId":"1:BASE_PARITY:d2ec268659721488"} 16:21:41 [16:21:41.875] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.875] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6f2a5f340fc053f4 err=test error {"provingJoId":"1:BASE_PARITY:6f2a5f340fc053f4"} 16:21:41 [16:21:41.876] INFO: prover-client:proving-broker Proving Broker started 16:21:41 [16:21:41.887] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3df1a4e41bf193a6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:3df1a4e41bf193a6"} 16:21:41 [16:21:41.888] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:3df1a4e41bf193a6 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:3df1a4e41bf193a6"} 16:21:41 [16:21:41.888] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3df1a4e41bf193a6 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:3df1a4e41bf193a6"} 16:21:41 [16:21:41.898] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:77e264d9965ad538 epochNumber=2 {"provingJobId":"2:BASE_PARITY:77e264d9965ad538"} 16:21:41 [16:21:41.899] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:77e264d9965ad538 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:77e264d9965ad538"} 16:21:41 [16:21:41.899] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:77e264d9965ad538 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:77e264d9965ad538"} 16:21:41 [16:21:41.961] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:2bc811c352c3f10c epochNumber=3 {"provingJobId":"3:BASE_PARITY:2bc811c352c3f10c"} 16:21:41 [16:21:41.977] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:42 [16:21:42.023] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:2c4c3b45388ca9fe epochNumber=4 {"provingJobId":"4:BASE_PARITY:2c4c3b45388ca9fe"} 16:21:42 [16:21:42.029] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:42 [16:21:42.085] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:9fe50109dbafacbe epochNumber=5 {"provingJobId":"5:BASE_PARITY:9fe50109dbafacbe"} 16:21:42 [16:21:42.105] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:42 [16:21:42.145] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.145] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6d07fa306e244ae5 epochNumber=42 {"provingJobId":"1:BASE_PARITY:6d07fa306e244ae5"} 16:21:42 [16:21:42.148] INFO: prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testbP8pmi/42 with map size 1073741824 16:21:42 [16:21:42.152] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.189] WARN: prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:73fc632c2517f400 epochNumber=1 {"provingJobId":"1:BASE_PARITY:73fc632c2517f400"} 16:21:42 [16:21:42.195] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.196] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:03f646d866569204 epochNumber=1 {"provingJobId":"1:BASE_PARITY:03f646d866569204"} 16:21:42 [16:21:42.196] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testEDHwyo/1 with map size 1073741824 16:21:42 [16:21:42.199] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.224] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fcc246aa71b306e6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:fcc246aa71b306e6"} 16:21:42 [16:21:42.235] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.236] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f9f85a054c18af54 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f9f85a054c18af54"} 16:21:42 [16:21:42.239] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testgISceY/1 with map size 1073741824 16:21:42 [16:21:42.244] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.268] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:f9f85a054c18af54 epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:f9f85a054c18af54"} 16:21:42 [16:21:42.274] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.275] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:773bff8afb97d02c epochNumber=1 {"provingJobId":"1:BASE_PARITY:773bff8afb97d02c"} 16:21:42 [16:21:42.277] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrlpbAH/1 with map size 1073741824 16:21:42 [16:21:42.281] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.302] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:773bff8afb97d02c epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:773bff8afb97d02c"} 16:21:42 [16:21:42.302] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:773bff8afb97d02c type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:773bff8afb97d02c"} 16:21:42 [16:21:42.309] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:773bff8afb97d02c epochNumber=1. Not enqueuing again {"provingJobId":"1:BASE_PARITY:773bff8afb97d02c"} 16:21:42 [16:21:42.312] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.313] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a42ea36b2435f0d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a42ea36b2435f0d6"} 16:21:42 [16:21:42.314] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8xNlZk/1 with map size 1073741824 16:21:42 [16:21:42.318] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.345] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.349] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.349] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:98abd26c5578bd41 epochNumber=1 {"provingJobId":"1:BASE_PARITY:98abd26c5578bd41"} 16:21:42 [16:21:42.350] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdqRbwW/1 with map size 1073741824 16:21:42 [16:21:42.352] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.370] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:98abd26c5578bd41 {"provingJobId":"1:BASE_PARITY:98abd26c5578bd41"} 16:21:42 [16:21:42.371] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:98abd26c5578bd41 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:98abd26c5578bd41"} 16:21:42 [16:21:42.371] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:98abd26c5578bd41 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:98abd26c5578bd41"} 16:21:42 [16:21:42.379] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.380] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f9a8033ca2908ec2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:f9a8033ca2908ec2"} 16:21:42 [16:21:42.381] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0i38fh/1 with map size 1073741824 16:21:42 [16:21:42.382] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.403] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f9a8033ca2908ec2 {"provingJobId":"1:BASE_PARITY:f9a8033ca2908ec2"} 16:21:42 [16:21:42.403] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f9a8033ca2908ec2 type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:f9a8033ca2908ec2"} 16:21:42 [16:21:42.413] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.413] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8b992b5ebdfa4218 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8b992b5ebdfa4218"} 16:21:42 [16:21:42.414] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFOJDiQ/1 with map size 1073741824 16:21:42 [16:21:42.415] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.431] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8b992b5ebdfa4218 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:8b992b5ebdfa4218"} 16:21:42 [16:21:42.431] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8b992b5ebdfa4218 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:8b992b5ebdfa4218"} 16:21:42 [16:21:42.439] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.440] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4f80f39a31573daa epochNumber=1 {"provingJobId":"1:BASE_PARITY:4f80f39a31573daa"} 16:21:42 [16:21:42.440] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test1uj3EP/1 with map size 1073741824 16:21:42 [16:21:42.442] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.461] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4f80f39a31573daa type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4f80f39a31573daa"} 16:21:42 [16:21:42.461] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4f80f39a31573daa type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4f80f39a31573daa"} 16:21:42 [16:21:42.473] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.474] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:7c044c0d0daf87db epochNumber=0 {"provingJobId":"1:BASE_PARITY:7c044c0d0daf87db"} 16:21:42 [16:21:42.476] INFO: prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-test8wTXS4/0 with map size 1073741824 16:21:42 [16:21:42.479] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.497] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7c044c0d0daf87db epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:7c044c0d0daf87db"} 16:21:42 [16:21:42.498] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5e12491294dfd86d epochNumber=0 {"provingJobId":"1:BASE_PARITY:5e12491294dfd86d"} 16:21:42 [16:21:42.498] WARN: prover-client:proving-broker Cached proving job id=1:BASE_PARITY:7c044c0d0daf87db epochNumber=0. Not enqueuing again {"provingJobId":"1:BASE_PARITY:7c044c0d0daf87db"} 16:21:42 [16:21:42.499] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e9d1bc420c26ef85 epochNumber=0 {"provingJobId":"1:BASE_PARITY:e9d1bc420c26ef85"} 16:21:42 [16:21:42.518] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.525] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.526] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:886ac9f111dbedf6 epochNumber=2 {"provingJobId":"1:BASE_PARITY:886ac9f111dbedf6"} 16:21:42 [16:21:42.527] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testQAEGvm/2 with map size 1073741824 16:21:42 [16:21:42.529] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.546] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:55af8edff3687256 epochNumber=1 {"provingJobId":"1:BASE_PARITY:55af8edff3687256"} 16:21:42 [16:21:42.547] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQAEGvm/1 with map size 1073741824 16:21:42 [16:21:42.549] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.567] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.567] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9da13021e27d18b6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9da13021e27d18b6"} 16:21:42 [16:21:42.568] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQGeGmn/1 with map size 1073741824 16:21:42 [16:21:42.570] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.589] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.590] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9298418accb35c04 epochNumber=1 {"provingJobId":"1:BASE_PARITY:9298418accb35c04"} 16:21:42 [16:21:42.591] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdXZB7a/1 with map size 1073741824 16:21:42 [16:21:42.592] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.610] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:070d00227003511c epochNumber=1 {"provingJobId":"1:BASE_PARITY:070d00227003511c"} 16:21:42 [16:21:42.620] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1c08ccd530ee010b epochNumber=2 {"provingJobId":"1:BASE_PARITY:1c08ccd530ee010b"} 16:21:42 [16:21:42.621] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testdXZB7a/2 with map size 1073741824 16:21:42 [16:21:42.628] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.647] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:14cf7f4d3b296acd epochNumber=1 {"provingJobId":"1:BASE_PARITY:14cf7f4d3b296acd"} 16:21:42 [16:21:42.658] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.659] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1bcf0e29db318fbb epochNumber=1 {"provingJobId":"1:BASE_PARITY:1bcf0e29db318fbb"} 16:21:42 [16:21:42.659] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwyhWrQ/1 with map size 1073741824 16:21:42 [16:21:42.662] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.681] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:3492d39a85bc2c5b epochNumber=1 {"provingJobId":"1:BASE_PARITY:3492d39a85bc2c5b"} 16:21:42 [16:21:42.687] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8630a2f113af3298 epochNumber=2 {"provingJobId":"1:BASE_PARITY:8630a2f113af3298"} 16:21:42 [16:21:42.688] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwyhWrQ/2 with map size 1073741824 16:21:42 [16:21:42.691] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.711] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8d67c70c92485f9e epochNumber=1 {"provingJobId":"1:BASE_PARITY:8d67c70c92485f9e"} 16:21:42 [16:21:42.722] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.723] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:8c9373050c4b6bd7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:8c9373050c4b6bd7"} 16:21:42 [16:21:42.723] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testT8WhCw/1 with map size 1073741824 16:21:42 [16:21:42.725] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.745] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:c900ef67192690a0 epochNumber=1 {"provingJobId":"1:BASE_PARITY:c900ef67192690a0"} 16:21:42 [16:21:42.751] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:803b8cb1320e8a22 epochNumber=2 {"provingJobId":"1:BASE_PARITY:803b8cb1320e8a22"} 16:21:42 [16:21:42.752] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testT8WhCw/2 with map size 1073741824 16:21:42 [16:21:42.753] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.770] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:fc4f26bd143dbbb3 epochNumber=1 {"provingJobId":"1:BASE_PARITY:fc4f26bd143dbbb3"} 16:21:42 [16:21:42.778] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.779] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cd4a940aa61fc99a epochNumber=1 {"provingJobId":"1:BASE_PARITY:cd4a940aa61fc99a"} 16:21:42 [16:21:42.779] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwNpwSA/1 with map size 1073741824 16:21:42 [16:21:42.780] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.797] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:dda84bf5a1c77113 epochNumber=1 {"provingJobId":"1:BASE_PARITY:dda84bf5a1c77113"} 16:21:42 [16:21:42.803] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cd4a940aa61fc99a type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cd4a940aa61fc99a"} 16:21:42 [16:21:42.810] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.811] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:34c5598db74011ad epochNumber=1 {"provingJobId":"1:BASE_PARITY:34c5598db74011ad"} 16:21:42 [16:21:42.813] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYpWz93/1 with map size 1073741824 16:21:42 [16:21:42.814] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.830] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:489c4060193a3c22 epochNumber=1 {"provingJobId":"1:BASE_PARITY:489c4060193a3c22"} 16:21:42 [16:21:42.835] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:34c5598db74011ad type=BASE_PARITY totalAttempts=1 err=result {"provingJobId":"1:BASE_PARITY:34c5598db74011ad"} 16:21:42 [16:21:42.844] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.844] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:292d9941616a2e28 epochNumber=1 {"provingJobId":"1:BASE_PARITY:292d9941616a2e28"} 16:21:42 [16:21:42.845] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test8PKmYq/1 with map size 1073741824 16:21:42 [16:21:42.846] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.864] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6f8c2cac9a8d65d6 epochNumber=1 {"provingJobId":"1:BASE_PARITY:6f8c2cac9a8d65d6"} 16:21:42 [16:21:42.869] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:292d9941616a2e28 type=BASE_PARITY retry=1 err=result {"provingJobId":"1:BASE_PARITY:292d9941616a2e28"} 16:21:42 [16:21:42.871] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.872] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:bffe017c57f4017a epochNumber=1 {"provingJobId":"1:BASE_PARITY:bffe017c57f4017a"} 16:21:42 [16:21:42.872] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testg0JDDz/1 with map size 1073741824 16:21:42 [16:21:42.873] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.893] INFO: prover-client:proving-broker Cancelling job id=1:BASE_PARITY:bffe017c57f4017a {"provingJobId":"1:BASE_PARITY:bffe017c57f4017a"} 16:21:42 [16:21:42.894] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:bffe017c57f4017a type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"1:BASE_PARITY:bffe017c57f4017a"} 16:21:42 [16:21:42.902] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:23fb73b879939f05 epochNumber=1 {"provingJobId":"1:BASE_PARITY:23fb73b879939f05"} 16:21:42 [16:21:42.907] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:bffe017c57f4017a has already been completed {"provingJobId":"1:BASE_PARITY:bffe017c57f4017a"} 16:21:42 [16:21:42.910] INFO: prover-client:proving-broker Proving Broker started 16:21:42 [16:21:42.910] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:02dc5ee89e9d2efe epochNumber=1 {"provingJobId":"1:BASE_PARITY:02dc5ee89e9d2efe"} 16:21:42 [16:21:42.911] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testL9MyYs/1 with map size 1073741824 16:21:42 [16:21:42.914] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:42 [16:21:42.930] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:64f7182a89342abc epochNumber=2 {"provingJobId":"1:BASE_PARITY:64f7182a89342abc"} 16:21:42 [16:21:42.931] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testL9MyYs/2 with map size 1073741824 16:21:42 [16:21:42.932] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:43 [16:21:43.998] INFO: prover-client:proving-broker Proving Broker started 16:21:44 [16:21:44.005] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:02dc5ee89e9d2efe settled=false {"provingJobId":"1:BASE_PARITY:02dc5ee89e9d2efe","status":"pending"} 16:21:44 [16:21:44.006] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:64f7182a89342abc settled=false {"provingJobId":"1:BASE_PARITY:64f7182a89342abc","status":"pending"} 16:21:44 [16:21:44.007] INFO: prover-client:proving-broker Proving job id=1:BASE_PARITY:02dc5ee89e9d2efe type=BASE_PARITY startedAt=1750090902947 older agent has taken job {"provingJobId":"1:BASE_PARITY:02dc5ee89e9d2efe"} 16:21:44 [16:21:44.008] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:02dc5ee89e9d2efe type=BASE_PARITY already being worked on by another agent. Sending new one {"provingJobId":"1:BASE_PARITY:02dc5ee89e9d2efe"} 16:21:44 [16:21:44.010] INFO: prover-client:proving-broker Proving Broker started 16:21:44 [16:21:44.011] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:604629e99d93c2a8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:604629e99d93c2a8"} 16:21:44 [16:21:44.011] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testWAsUWq/1 with map size 1073741824 16:21:44 [16:21:44.013] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:44 [16:21:44.031] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5006fbaa26d47009 epochNumber=2 {"provingJobId":"1:BASE_PARITY:5006fbaa26d47009"} 16:21:44 [16:21:44.032] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testWAsUWq/2 with map size 1073741824 16:21:44 [16:21:44.033] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:45 [16:21:45.050] INFO: prover-client:proving-broker Proving Broker started 16:21:45 [16:21:45.051] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:604629e99d93c2a8 settled=false {"provingJobId":"1:BASE_PARITY:604629e99d93c2a8","status":"pending"} 16:21:45 [16:21:45.053] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5006fbaa26d47009 settled=false {"provingJobId":"1:BASE_PARITY:5006fbaa26d47009","status":"pending"} 16:21:45 [16:21:45.054] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:604629e99d93c2a8 type=BASE_PARITY not found in the in-progress cache, adding it {"provingJobId":"1:BASE_PARITY:604629e99d93c2a8"} 16:21:45 [16:21:45.061] INFO: prover-client:proving-broker Proving Broker started 16:21:45 [16:21:45.062] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cac4cf5c109526af epochNumber=1 {"provingJobId":"1:BASE_PARITY:cac4cf5c109526af"} 16:21:45 [16:21:45.063] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test98MzzF/1 with map size 1073741824 16:21:45 [16:21:45.066] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:45 [16:21:45.086] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6ab0a4ec00c74f9c epochNumber=2 {"provingJobId":"1:BASE_PARITY:6ab0a4ec00c74f9c"} 16:21:45 [16:21:45.087] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test98MzzF/2 with map size 1073741824 16:21:45 [16:21:45.091] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.109] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.112] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:cac4cf5c109526af settled=false {"provingJobId":"1:BASE_PARITY:cac4cf5c109526af","status":"pending"} 16:21:46 [16:21:46.114] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6ab0a4ec00c74f9c settled=false {"provingJobId":"1:BASE_PARITY:6ab0a4ec00c74f9c","status":"pending"} 16:21:46 [16:21:46.116] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:cac4cf5c109526af type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:cac4cf5c109526af"} 16:21:46 [16:21:46.116] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:cac4cf5c109526af type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:cac4cf5c109526af"} 16:21:46 [16:21:46.128] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.128] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0c2270d9848d676b epochNumber=1 {"provingJobId":"1:BASE_PARITY:0c2270d9848d676b"} 16:21:46 [16:21:46.130] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSoqFGz/1 with map size 1073741824 16:21:46 [16:21:46.133] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.148] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:aae25421a2011ef6 epochNumber=2 {"provingJobId":"1:BASE_PARITY:aae25421a2011ef6"} 16:21:46 [16:21:46.150] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testSoqFGz/2 with map size 1073741824 16:21:46 [16:21:46.152] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.170] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0c2270d9848d676b type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0c2270d9848d676b"} 16:21:46 [16:21:46.177] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:aae25421a2011ef6 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:aae25421a2011ef6"} 16:21:46 [16:21:46.186] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.187] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:0140272136a0c5b7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:0140272136a0c5b7"} 16:21:46 [16:21:46.187] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testclRwhP/1 with map size 1073741824 16:21:46 [16:21:46.190] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.207] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:9e8835a935991649 epochNumber=2 {"provingJobId":"1:BASE_PARITY:9e8835a935991649"} 16:21:46 [16:21:46.208] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testclRwhP/2 with map size 1073741824 16:21:46 [16:21:46.210] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.237] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:0140272136a0c5b7 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:0140272136a0c5b7"} 16:21:46 [16:21:46.238] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:0140272136a0c5b7 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:0140272136a0c5b7"} 16:21:46 [16:21:46.245] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9e8835a935991649 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:9e8835a935991649"} 16:21:46 [16:21:46.255] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.256] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:c1e5c7233cb61689 err=test error {"provingJoId":"1:BASE_PARITY:c1e5c7233cb61689"} 16:21:46 [16:21:46.259] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.259] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:66d5792b4afca468 not found {"provingJobId":"1:BASE_PARITY:66d5792b4afca468"} 16:21:46 [16:21:46.261] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.262] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1636d013d1e2f3d8 epochNumber=1 {"provingJobId":"1:BASE_PARITY:1636d013d1e2f3d8"} 16:21:46 [16:21:46.262] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testybDhU8/1 with map size 1073741824 16:21:46 [16:21:46.266] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.292] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.293] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:e0973e289431a00a epochNumber=1 {"provingJobId":"1:BASE_PARITY:e0973e289431a00a"} 16:21:46 [16:21:46.293] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPUIwOg/1 with map size 1073741824 16:21:46 [16:21:46.297] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.421] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:e0973e289431a00a timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:e0973e289431a00a"} 16:21:46 [16:21:46.427] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.428] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:acc33300a93102f2 epochNumber=1 {"provingJobId":"1:BASE_PARITY:acc33300a93102f2"} 16:21:46 [16:21:46.428] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testc7WMiu/1 with map size 1073741824 16:21:46 [16:21:46.431] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.554] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:acc33300a93102f2 timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:acc33300a93102f2"} 16:21:46 [16:21:46.559] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1443a0e973a4383d epochNumber=10 {"provingJobId":"1:BASE_PARITY:1443a0e973a4383d"} 16:21:46 [16:21:46.560] INFO: prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testc7WMiu/10 with map size 1073741824 16:21:46 [16:21:46.562] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.605] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:46 [16:21:46.605] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:21:46 [16:21:46.709] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.710] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:782267dcf3de7eda epochNumber=1 {"provingJobId":"1:BASE_PARITY:782267dcf3de7eda"} 16:21:46 [16:21:46.710] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJa0a64/1 with map size 1073741824 16:21:46 [16:21:46.712] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.886] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:782267dcf3de7eda timed out. Adding it back to the queue. {"provingJobId":"1:BASE_PARITY:782267dcf3de7eda"} 16:21:46 [16:21:46.942] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.943] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:343b058a360fddef epochNumber=1 {"provingJobId":"1:BASE_PARITY:343b058a360fddef"} 16:21:46 [16:21:46.945] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOfUdg7/1 with map size 1073741824 16:21:46 [16:21:46.948] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.967] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:343b058a360fddef type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:343b058a360fddef"} 16:21:46 [16:21:46.971] INFO: prover-client:proving-broker Proving Broker started 16:21:46 [16:21:46.972] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:57fb2b1c472c6ef4 epochNumber=1 {"provingJobId":"1:BASE_PARITY:57fb2b1c472c6ef4"} 16:21:46 [16:21:46.974] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5ElBpI/1 with map size 1073741824 16:21:46 [16:21:46.978] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:46 [16:21:46.997] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:57fb2b1c472c6ef4 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:57fb2b1c472c6ef4"} 16:21:46 [16:21:46.998] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:57fb2b1c472c6ef4 type=BASE_PARITY totalAttempts=2 err=test error {"provingJobId":"1:BASE_PARITY:57fb2b1c472c6ef4"} 16:21:47 [16:21:47.008] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.008] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:1f161d4631c9397b epochNumber=1 {"provingJobId":"1:BASE_PARITY:1f161d4631c9397b"} 16:21:47 [16:21:47.009] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbfECMV/1 with map size 1073741824 16:21:47 [16:21:47.017] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.037] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1f161d4631c9397b type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:1f161d4631c9397b"} 16:21:47 [16:21:47.047] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.048] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cb445293777bf8a7 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cb445293777bf8a7"} 16:21:47 [16:21:47.049] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test4PfE7F/1 with map size 1073741824 16:21:47 [16:21:47.052] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.071] INFO: prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:cb445293777bf8a7 type=BASE_PARITY retry=1 err=test error {"provingJobId":"1:BASE_PARITY:cb445293777bf8a7"} 16:21:47 [16:21:47.073] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:094481d8bac80d9d epochNumber=3 {"provingJobId":"1:BASE_PARITY:094481d8bac80d9d"} 16:21:47 [16:21:47.073] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-test4PfE7F/3 with map size 1073741824 16:21:47 [16:21:47.076] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.098] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:47 [16:21:47.099] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:21:47 [16:21:47.120] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:cb445293777bf8a7 err=test error {"provingJoId":"1:BASE_PARITY:cb445293777bf8a7"} 16:21:47 [16:21:47.129] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3qix90/1 with map size 1073741824 16:21:47 [16:21:47.133] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.156] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test3qix90/2 with map size 1073741824 16:21:47 [16:21:47.160] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.179] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.182] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:33bc9d4ad04522e0 settled=false {"provingJobId":"1:BASE_PARITY:33bc9d4ad04522e0","status":"pending"} 16:21:47 [16:21:47.186] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:92f8bb1bcf15a40a settled=false {"provingJobId":"1:BASE_PARITY:92f8bb1bcf15a40a","status":"pending"} 16:21:47 [16:21:47.191] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testG0UAOH/1 with map size 1073741824 16:21:47 [16:21:47.195] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.222] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testG0UAOH/2 with map size 1073741824 16:21:47 [16:21:47.228] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.255] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.259] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:b1ae8db7f0d424d1 settled=true {"provingJobId":"1:BASE_PARITY:b1ae8db7f0d424d1","status":"fulfilled"} 16:21:47 [16:21:47.262] INFO: prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:6fad9276a3921bbf settled=true {"provingJobId":"2:BASE_PARITY:6fad9276a3921bbf","status":"fulfilled"} 16:21:47 [16:21:47.268] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZirC1f/1 with map size 1073741824 16:21:47 [16:21:47.271] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.298] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZirC1f/2 with map size 1073741824 16:21:47 [16:21:47.300] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.320] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.323] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:e27970cf7f9d6429 settled=true {"provingJobId":"1:BASE_PARITY:e27970cf7f9d6429","status":"fulfilled"} 16:21:47 [16:21:47.325] INFO: prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:bf1d96b8fa1b943e settled=false {"provingJobId":"1:BASE_PARITY:bf1d96b8fa1b943e","status":"pending"} 16:21:47 [16:21:47.331] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.331] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:947b6de02527c56b epochNumber=1 {"provingJobId":"1:BASE_PARITY:947b6de02527c56b"} 16:21:47 [16:21:47.332] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testyORAWv/1 with map size 1073741824 16:21:47 [16:21:47.335] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.361] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.362] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:cdd8cb253924d758 epochNumber=1 {"provingJobId":"1:BASE_PARITY:cdd8cb253924d758"} 16:21:47 [16:21:47.363] ERROR: prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:cdd8cb253924d758: Error: db error: Error: db error 16:21:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63) {"provingJobId":"1:BASE_PARITY:cdd8cb253924d758"} 16:21:47 [16:21:47.392] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.395] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:efeb370d14143993 epochNumber=1 {"provingJobId":"1:BASE_PARITY:efeb370d14143993"} 16:21:47 [16:21:47.398] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrWW2qH/1 with map size 1073741824 16:21:47 [16:21:47.401] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.425] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:efeb370d14143993 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:efeb370d14143993"} 16:21:47 [16:21:47.426] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:efeb370d14143993 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:efeb370d14143993"} 16:21:47 [16:21:47.444] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.445] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:83ebbccc2e109e12 epochNumber=1 {"provingJobId":"1:BASE_PARITY:83ebbccc2e109e12"} 16:21:47 [16:21:47.448] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9WRjEO/1 with map size 1073741824 16:21:47 [16:21:47.453] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.474] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:83ebbccc2e109e12 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:83ebbccc2e109e12"} 16:21:47 [16:21:47.475] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:83ebbccc2e109e12 type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:83ebbccc2e109e12"} 16:21:47 [16:21:47.477] ERROR: prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:83ebbccc2e109e12: Error: db error 16:21:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69) {"provingJobId":"1:BASE_PARITY:83ebbccc2e109e12"} 16:21:47 [16:21:47.484] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.485] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4f805164066dbe63 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4f805164066dbe63"} 16:21:47 [16:21:47.486] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsNXEqk/1 with map size 1073741824 16:21:47 [16:21:47.492] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.548] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:4f805164066dbe63 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:4f805164066dbe63"} 16:21:47 [16:21:47.549] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:4f805164066dbe63 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:4f805164066dbe63"} 16:21:47 [16:21:47.563] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.564] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:b4beab86be2f0e64 epochNumber=1 {"provingJobId":"1:BASE_PARITY:b4beab86be2f0e64"} 16:21:47 [16:21:47.565] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzi8ljm/1 with map size 1073741824 16:21:47 [16:21:47.571] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.592] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:b4beab86be2f0e64 type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:b4beab86be2f0e64"} 16:21:47 [16:21:47.592] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b4beab86be2f0e64 type=BASE_PARITY totalAttempts=1 err=test error {"provingJobId":"1:BASE_PARITY:b4beab86be2f0e64"} 16:21:47 [16:21:47.593] ERROR: prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:b4beab86be2f0e64 jobErr=test error: Error: db error 16:21:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68) {"provingJobId":"1:BASE_PARITY:b4beab86be2f0e64"} 16:21:47 [16:21:47.603] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.604] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:8c61c2f03e636721 not found {"provingJobId":"1:BASE_PARITY:8c61c2f03e636721"} 16:21:47 [16:21:47.613] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.614] WARN: prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:37893c6540a5c73d err=test error {"provingJoId":"1:BASE_PARITY:37893c6540a5c73d"} 16:21:47 [16:21:47.628] INFO: prover-client:proving-broker Proving Broker started 16:21:47 [16:21:47.639] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:d6dc5dd22053ceca epochNumber=1 {"provingJobId":"1:BASE_PARITY:d6dc5dd22053ceca"} 16:21:47 [16:21:47.641] INFO: prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testnrZAWY/1 with map size 1073741824 16:21:47 [16:21:47.646] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.673] WARN: prover-client:proving-broker Proving job id=1:BASE_PARITY:d6dc5dd22053ceca type=BASE_PARITY not in the in-progress set {"provingJobId":"1:BASE_PARITY:d6dc5dd22053ceca"} 16:21:47 [16:21:47.674] INFO: prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d6dc5dd22053ceca type=BASE_PARITY totalAttempts=1 {"provingJobId":"1:BASE_PARITY:d6dc5dd22053ceca"} 16:21:47 [16:21:47.692] INFO: prover-client:proving-broker New proving job id=2:BASE_PARITY:daf8f98f3dd82ea5 epochNumber=2 {"provingJobId":"2:BASE_PARITY:daf8f98f3dd82ea5"} 16:21:47 [16:21:47.695] INFO: prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testnrZAWY/2 with map size 1073741824 16:21:47 [16:21:47.698] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.721] WARN: prover-client:proving-broker Proving job id=2:BASE_PARITY:daf8f98f3dd82ea5 type=BASE_PARITY not in the in-progress set {"provingJobId":"2:BASE_PARITY:daf8f98f3dd82ea5"} 16:21:47 [16:21:47.722] INFO: prover-client:proving-broker Proving job complete id=2:BASE_PARITY:daf8f98f3dd82ea5 type=BASE_PARITY totalAttempts=1 {"provingJobId":"2:BASE_PARITY:daf8f98f3dd82ea5"} 16:21:47 [16:21:47.793] INFO: prover-client:proving-broker New proving job id=3:BASE_PARITY:6a6332a3911bdb7b epochNumber=3 {"provingJobId":"3:BASE_PARITY:6a6332a3911bdb7b"} 16:21:47 [16:21:47.795] INFO: prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testnrZAWY/3 with map size 1073741824 16:21:47 [16:21:47.798] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.833] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:47 [16:21:47.834] INFO: prover-client:proving-broker-database Deleting broker database for epoch 1 16:21:47 [16:21:47.885] INFO: prover-client:proving-broker New proving job id=4:BASE_PARITY:55afc1bf627bb399 epochNumber=4 {"provingJobId":"4:BASE_PARITY:55afc1bf627bb399"} 16:21:47 [16:21:47.887] INFO: prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testnrZAWY/4 with map size 1073741824 16:21:47 [16:21:47.888] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:47 [16:21:47.912] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:47 [16:21:47.912] INFO: prover-client:proving-broker-database Deleting broker database for epoch 2 16:21:47 [16:21:47.966] INFO: prover-client:proving-broker New proving job id=5:BASE_PARITY:0bb2e6cb7ab2ac9f epochNumber=5 {"provingJobId":"5:BASE_PARITY:0bb2e6cb7ab2ac9f"} 16:21:47 [16:21:47.967] INFO: prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testnrZAWY/5 with map size 1073741824 16:21:47 [16:21:47.970] INFO: kv-store:lmdb-v2 Starting data store with maxReaders 16 16:21:48 [16:21:48.017] INFO: prover-client:proving-broker Cleaned up jobs=1 16:21:48 [16:21:48.017] INFO: prover-client:proving-broker-database Deleting broker database for epoch 3 16:21:48 PASS src/proving_broker/proving_broker.test.ts (12.6 s) 16:21:48 ProvingBroker 16:21:48 Producer API 16:21:48 ✓ refuses stale jobs (39 ms) 16:21:48 ✓ enqueues jobs (3 ms) 16:21:48 ✓ ignores duplicate jobs (2 ms) 16:21:48 ✓ reports correct status when enqueuing repeat jobs (3 ms) 16:21:48 ✓ throws an error in case of duplicate job IDs (18 ms) 16:21:48 ✓ returns not-found status for non-existing jobs (2 ms) 16:21:48 ✓ cancels jobs in queue (3 ms) 16:21:48 ✓ cancels jobs in-progress (2 ms) 16:21:48 ✓ returns job result if successful (2 ms) 16:21:48 ✓ returns job error if failed (1 ms) 16:21:48 ✓ correctly returns job status for concurrent writes (2 ms) 16:21:48 ✓ refuses stale jobs (55 ms) 16:21:48 ✓ enqueues jobs (40 ms) 16:21:48 ✓ ignores duplicate jobs (38 ms) 16:21:48 ✓ reports correct status when enqueuing repeat jobs (39 ms) 16:21:48 ✓ throws an error in case of duplicate job IDs (32 ms) 16:21:48 ✓ returns not-found status for non-existing jobs (3 ms) 16:21:48 ✓ cancels jobs in queue (31 ms) 16:21:48 ✓ cancels jobs in-progress (33 ms) 16:21:48 ✓ returns job result if successful (27 ms) 16:21:48 ✓ returns job error if failed (32 ms) 16:21:48 ✓ correctly returns job status for concurrent writes (42 ms) 16:21:48 Consumer API 16:21:48 ✓ returns undefined if no jobs are available (1 ms) 16:21:48 ✓ returns jobs in priority order (1 ms) 16:21:48 ✓ returns undefined if no jobs are available for the given allowList (20 ms) 16:21:48 ✓ returns a job if it is in the allowList (3 ms) 16:21:48 ✓ returns the most important job if it is in the allowList (3 ms) 16:21:48 ✓ returns any job if filter is empty (2 ms) 16:21:48 ✓ returns a new job when reporting job success (2 ms) 16:21:48 ✓ returns a new job when reporting permanent error (2 ms) 16:21:48 ✓ returns a new job when reporting retry-able error (1 ms) 16:21:48 ✓ returns a new job when reporting progress if current one is cancelled (3 ms) 16:21:48 ✓ returns a new job if job is already in progress elsewhere (1056 ms) 16:21:48 ✓ avoids sending the same job to a new agent after a restart (1005 ms) 16:21:48 ✓ avoids sending a completed job to a new agent after a restart (1010 ms) 16:21:48 ✓ tracks job result if in progress (10 ms) 16:21:48 ✓ tracks job result even if job is in queue (4 ms) 16:21:48 ✓ ignores reported job error if unknown job (2 ms) 16:21:48 ✓ ignores job result if unknown job (2 ms) 16:21:48 ✓ returns undefined if no jobs are available (6 ms) 16:21:48 ✓ returns jobs in priority order (46 ms) 16:21:48 ✓ returns undefined if no jobs are available for the given allowList (22 ms) 16:21:48 ✓ returns a job if it is in the allowList (67 ms) 16:21:48 ✓ returns the most important job if it is in the allowList (65 ms) 16:21:48 ✓ returns any job if filter is empty (57 ms) 16:21:48 ✓ returns a new job when reporting job success (32 ms) 16:21:48 ✓ returns a new job when reporting permanent error (33 ms) 16:21:48 ✓ returns a new job when reporting retry-able error (28 ms) 16:21:48 ✓ returns a new job when reporting progress if current one is cancelled (38 ms) 16:21:48 ✓ returns a new job if job is already in progress elsewhere (1101 ms) 16:21:48 ✓ avoids sending the same job to a new agent after a restart (1048 ms) 16:21:48 ✓ avoids sending a completed job to a new agent after a restart (1068 ms) 16:21:48 ✓ tracks job result if in progress (58 ms) 16:21:48 ✓ tracks job result even if job is in queue (68 ms) 16:21:48 ✓ ignores reported job error if unknown job (5 ms) 16:21:48 ✓ ignores job result if unknown job (3 ms) 16:21:48 Timeouts 16:21:48 ✓ tracks in progress jobs (3 ms) 16:21:48 ✓ re-enqueues jobs that time out (109 ms) 16:21:48 ✓ cancel stale jobs that time out (254 ms) 16:21:48 ✓ keeps the jobs in progress while it is alive (205 ms) 16:21:48 ✓ tracks in progress jobs (29 ms) 16:21:48 ✓ re-enqueues jobs that time out (135 ms) 16:21:48 ✓ cancel stale jobs that time out (281 ms) 16:21:48 ✓ keeps the jobs in progress while it is alive (231 ms) 16:21:48 Retries 16:21:48 ✓ retries jobs (4 ms) 16:21:48 ✓ retries up to a maximum number of times (4 ms) 16:21:48 ✓ passing retry=false does not retry (2 ms) 16:21:48 ✓ does not retry if job is stale (28 ms) 16:21:48 ✓ retries jobs (31 ms) 16:21:48 ✓ retries up to a maximum number of times (35 ms) 16:21:48 ✓ passing retry=false does not retry (40 ms) 16:21:48 ✓ does not retry if job is stale (75 ms) 16:21:48 Database management 16:21:48 ✓ re-enqueues proof requests on start (3 ms) 16:21:48 ✓ restores proof results on start (1 ms) 16:21:48 ✓ only re-enqueues unfinished jobs (1 ms) 16:21:48 ✓ saves job when enqueued (3 ms) 16:21:48 ✓ does not retain job if database fails to save (3 ms) 16:21:48 ✓ saves job result (2 ms) 16:21:48 ✓ saves result even if database fails to save (2 ms) 16:21:48 ✓ saves job error (2 ms) 16:21:48 ✓ saves job error even if database fails to save (2 ms) 16:21:48 ✓ does not save job result if job is unknown (1 ms) 16:21:48 ✓ does not save job error if job is unknown (2 ms) 16:21:48 ✓ cleans up old jobs periodically (260 ms) 16:21:48 ✓ re-enqueues proof requests on start (67 ms) 16:21:48 ✓ restores proof results on start (75 ms) 16:21:48 ✓ only re-enqueues unfinished jobs (65 ms) 16:21:48 ✓ saves job when enqueued (28 ms) 16:21:48 ✓ does not retain job if database fails to save (6 ms) 16:21:48 ✓ saves job result (75 ms) 16:21:48 ✓ saves result even if database fails to save (38 ms) 16:21:48 ✓ saves job error (80 ms) 16:21:48 ✓ saves job error even if database fails to save (39 ms) 16:21:48 ✓ does not save job result if job is unknown (7 ms) 16:21:48 ✓ does not save job error if job is unknown (9 ms) 16:21:48 ✓ cleans up old jobs periodically (427 ms) 16:21:48 16:21:48 Test Suites: 1 passed, 1 total 16:21:48 Tests: 96 passed, 96 total 16:21:48 Snapshots: 0 total 16:21:48 Time: 12.694 s 16:21:48 Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i. 16:21:48 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?