Parent Log:
http://ci.aztec-labs.com/6a78b6e0a15351b6
Command: a5511b51e55ea9c0 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/6ee95286f8bcc5bb6dca43222459ac5bf43a8a75
Env: REF_NAME=gh-readonly-queue/next/pr-15156-9cbd161a1dfa928167c1f969200b95df55ed4c6b CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Fri Jun 20 09:43:30 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15156_amd64_x1-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
09:43:34 [09:43:34.422]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.428]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f8dde92b75b85295 epochNumber=42
{"provingJobId":"1:BASE_PARITY:f8dde92b75b85295"}
09:43:34 [09:43:34.431]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:58ec5a97935dbe1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:58ec5a97935dbe1f"}
09:43:34 [09:43:34.458]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.459]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87580bc927f0bee2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87580bc927f0bee2"}
09:43:34 [09:43:34.460]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7779c6c8c04236ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:7779c6c8c04236ea"}
09:43:34 [09:43:34.462]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.463]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b317551894c15eb9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b317551894c15eb9"}
09:43:34 [09:43:34.464]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b317551894c15eb9 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b317551894c15eb9"}
09:43:34 [09:43:34.466]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.467]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d32fe40fe66033b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d32fe40fe66033b8"}
09:43:34 [09:43:34.467]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d32fe40fe66033b8 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d32fe40fe66033b8"}
09:43:34 [09:43:34.468]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d32fe40fe66033b8 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d32fe40fe66033b8"}
09:43:34 [09:43:34.469]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d32fe40fe66033b8 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d32fe40fe66033b8"}
09:43:34 [09:43:34.470]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.470]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7993b4d944cb37ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:7993b4d944cb37ad"}
09:43:34 [09:43:34.491]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.492]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.493]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36057874c78fcd49 epochNumber=1
{"provingJobId":"1:BASE_PARITY:36057874c78fcd49"}
09:43:34 [09:43:34.493]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:36057874c78fcd49
{"provingJobId":"1:BASE_PARITY:36057874c78fcd49"}
09:43:34 [09:43:34.494]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:36057874c78fcd49 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:36057874c78fcd49"}
09:43:34 [09:43:34.494]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:36057874c78fcd49 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:36057874c78fcd49"}
09:43:34 [09:43:34.495]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.496]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c7b981b8cdee891a epochNumber=1
{"provingJobId":"1:BASE_PARITY:c7b981b8cdee891a"}
09:43:34 [09:43:34.497]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:c7b981b8cdee891a
{"provingJobId":"1:BASE_PARITY:c7b981b8cdee891a"}
09:43:34 [09:43:34.497]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:c7b981b8cdee891a type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:c7b981b8cdee891a"}
09:43:34 [09:43:34.499]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.499]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55adbcf7c7e90c1c epochNumber=1
{"provingJobId":"1:BASE_PARITY:55adbcf7c7e90c1c"}
09:43:34 [09:43:34.500]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:55adbcf7c7e90c1c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:55adbcf7c7e90c1c"}
09:43:34 [09:43:34.500]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:55adbcf7c7e90c1c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:55adbcf7c7e90c1c"}
09:43:34 [09:43:34.501]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.502]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48667d60f8770749 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48667d60f8770749"}
09:43:34 [09:43:34.503]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:48667d60f8770749 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:48667d60f8770749"}
09:43:34 [09:43:34.503]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:48667d60f8770749 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:48667d60f8770749"}
09:43:34 [09:43:34.504]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.505]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d8f0a60699ca0847 epochNumber=0
{"provingJobId":"1:BASE_PARITY:d8f0a60699ca0847"}
09:43:34 [09:43:34.505]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d8f0a60699ca0847 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d8f0a60699ca0847"}
09:43:34 [09:43:34.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:be97c3ef3cd18a55 epochNumber=0
{"provingJobId":"1:BASE_PARITY:be97c3ef3cd18a55"}
09:43:34 [09:43:34.506]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:d8f0a60699ca0847 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:d8f0a60699ca0847"}
09:43:34 [09:43:34.506]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7be6266d1ef6fced epochNumber=0
{"provingJobId":"1:BASE_PARITY:7be6266d1ef6fced"}
09:43:34 [09:43:34.516]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.518]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.519]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11777f70291dc8c1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:11777f70291dc8c1"}
09:43:34 [09:43:34.520]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:77e7ea6cdd8b8f86 epochNumber=1
{"provingJobId":"1:BASE_PARITY:77e7ea6cdd8b8f86"}
09:43:34 [09:43:34.521]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.522]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9b60c695da9c7360 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9b60c695da9c7360"}
09:43:34 [09:43:34.523]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.524]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a7e90fc7da0a544e epochNumber=1
{"provingJobId":"1:BASE_PARITY:a7e90fc7da0a544e"}
09:43:34 [09:43:34.524]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ec2dd65e466d8ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ec2dd65e466d8ea"}
09:43:34 [09:43:34.524]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5b82ba6f6a66727c epochNumber=2
{"provingJobId":"1:BASE_PARITY:5b82ba6f6a66727c"}
09:43:34 [09:43:34.525]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a0ba6f93dab3bc75 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a0ba6f93dab3bc75"}
09:43:34 [09:43:34.526]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9895013114823915 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9895013114823915"}
09:43:34 [09:43:34.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6617f44710c347f8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6617f44710c347f8"}
09:43:34 [09:43:34.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f66b8bd69dff9fc epochNumber=2
{"provingJobId":"1:BASE_PARITY:4f66b8bd69dff9fc"}
09:43:34 [09:43:34.527]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06bfbedc1e6e0534 epochNumber=1
{"provingJobId":"1:BASE_PARITY:06bfbedc1e6e0534"}
09:43:34 [09:43:34.529]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.529]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cfdcb1b054942785 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cfdcb1b054942785"}
09:43:34 [09:43:34.530]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87d85cf0f9941ef4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87d85cf0f9941ef4"}
09:43:34 [09:43:34.530]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b47d59b7c7d3016 epochNumber=2
{"provingJobId":"1:BASE_PARITY:1b47d59b7c7d3016"}
09:43:34 [09:43:34.530]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e548d6dbb7dc6d1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e548d6dbb7dc6d1"}
09:43:34 [09:43:34.532]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.532]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:08f6ac339cbe5e9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:08f6ac339cbe5e9f"}
09:43:34 [09:43:34.534]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4d79de25a9bcb0b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4d79de25a9bcb0b4"}
09:43:34 [09:43:34.534]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:08f6ac339cbe5e9f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:08f6ac339cbe5e9f"}
09:43:34 [09:43:34.536]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:715204026c7c1754 epochNumber=1
{"provingJobId":"1:BASE_PARITY:715204026c7c1754"}
09:43:34 [09:43:34.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2fd44833a6a4b509 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2fd44833a6a4b509"}
09:43:34 [09:43:34.538]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:715204026c7c1754 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:715204026c7c1754"}
09:43:34 [09:43:34.539]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.539]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e7145b8fdbbd1580 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e7145b8fdbbd1580"}
09:43:34 [09:43:34.540]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:56a9f9fc3e63c426 epochNumber=1
{"provingJobId":"1:BASE_PARITY:56a9f9fc3e63c426"}
09:43:34 [09:43:34.540]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e7145b8fdbbd1580 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:e7145b8fdbbd1580"}
09:43:34 [09:43:34.541]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.542]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6966cd03998eb988 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6966cd03998eb988"}
09:43:34 [09:43:34.542]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:6966cd03998eb988
{"provingJobId":"1:BASE_PARITY:6966cd03998eb988"}
09:43:34 [09:43:34.543]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6966cd03998eb988 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:6966cd03998eb988"}
09:43:34 [09:43:34.543]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc1229e7ec4c5fd5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc1229e7ec4c5fd5"}
09:43:34 [09:43:34.543]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6966cd03998eb988 has already been completed
{"provingJobId":"1:BASE_PARITY:6966cd03998eb988"}
09:43:34 [09:43:34.545]
INFO:
prover-client:proving-broker Proving Broker started
09:43:34 [09:43:34.545]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5fba4a865766e19b epochNumber=1
{"provingJobId":"1:BASE_PARITY:5fba4a865766e19b"}
09:43:34 [09:43:34.545]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21ecb1c36eaeb46b epochNumber=2
{"provingJobId":"1:BASE_PARITY:21ecb1c36eaeb46b"}
09:43:35 [09:43:35.598]
INFO:
prover-client:proving-broker Proving Broker started
09:43:35 [09:43:35.599]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5fba4a865766e19b settled=false
{"provingJobId":"1:BASE_PARITY:5fba4a865766e19b","status":"pending"}
09:43:35 [09:43:35.599]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:21ecb1c36eaeb46b settled=false
{"provingJobId":"1:BASE_PARITY:21ecb1c36eaeb46b","status":"pending"}
09:43:35 [09:43:35.600]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5fba4a865766e19b type=BASE_PARITY startedAt=1750412614546 older agent has taken job
{"provingJobId":"1:BASE_PARITY:5fba4a865766e19b"}
09:43:35 [09:43:35.601]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5fba4a865766e19b type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:5fba4a865766e19b"}
09:43:35 [09:43:35.602]
INFO:
prover-client:proving-broker Proving Broker started
09:43:35 [09:43:35.603]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:11aa1de1a118220f epochNumber=1
{"provingJobId":"1:BASE_PARITY:11aa1de1a118220f"}
09:43:35 [09:43:35.603]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0c77c8e153fe8351 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0c77c8e153fe8351"}
09:43:36 [09:43:36.604]
INFO:
prover-client:proving-broker Proving Broker started
09:43:36 [09:43:36.605]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:11aa1de1a118220f settled=false
{"provingJobId":"1:BASE_PARITY:11aa1de1a118220f","status":"pending"}
09:43:36 [09:43:36.605]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0c77c8e153fe8351 settled=false
{"provingJobId":"1:BASE_PARITY:0c77c8e153fe8351","status":"pending"}
09:43:36 [09:43:36.606]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:11aa1de1a118220f type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:11aa1de1a118220f"}
09:43:36 [09:43:36.608]
INFO:
prover-client:proving-broker Proving Broker started
09:43:36 [09:43:36.608]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4cd321696e32f845 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4cd321696e32f845"}
09:43:36 [09:43:36.609]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3bf59e7e8306be92 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3bf59e7e8306be92"}
09:43:37 [09:43:37.612]
INFO:
prover-client:proving-broker Proving Broker started
09:43:37 [09:43:37.613]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4cd321696e32f845 settled=false
{"provingJobId":"1:BASE_PARITY:4cd321696e32f845","status":"pending"}
09:43:37 [09:43:37.614]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3bf59e7e8306be92 settled=false
{"provingJobId":"1:BASE_PARITY:3bf59e7e8306be92","status":"pending"}
09:43:37 [09:43:37.615]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4cd321696e32f845 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4cd321696e32f845"}
09:43:37 [09:43:37.615]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4cd321696e32f845 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4cd321696e32f845"}
09:43:37 [09:43:37.617]
INFO:
prover-client:proving-broker Proving Broker started
09:43:37 [09:43:37.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d5b6f654d659cb24 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d5b6f654d659cb24"}
09:43:37 [09:43:37.618]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3996bac43cdd2e90 epochNumber=2
{"provingJobId":"1:BASE_PARITY:3996bac43cdd2e90"}
09:43:37 [09:43:37.619]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d5b6f654d659cb24 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d5b6f654d659cb24"}
09:43:37 [09:43:37.620]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3996bac43cdd2e90 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:3996bac43cdd2e90"}
09:43:37 [09:43:37.621]
INFO:
prover-client:proving-broker Proving Broker started
09:43:37 [09:43:37.622]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ecca521ee6a66db1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ecca521ee6a66db1"}
09:43:37 [09:43:37.622]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b00d6c50845f7c59 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b00d6c50845f7c59"}
09:43:37 [09:43:37.622]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ecca521ee6a66db1 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ecca521ee6a66db1"}
09:43:37 [09:43:37.622]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ecca521ee6a66db1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ecca521ee6a66db1"}
09:43:37 [09:43:37.623]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b00d6c50845f7c59 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b00d6c50845f7c59"}
09:43:37 [09:43:37.624]
INFO:
prover-client:proving-broker Proving Broker started
09:43:37 [09:43:37.624]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:3f95cb48affae6a9 err=test error
{"provingJoId":"1:BASE_PARITY:3f95cb48affae6a9"}
09:43:37 [09:43:37.625]
INFO:
prover-client:proving-broker Proving Broker started
09:43:37 [09:43:37.626]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:299f9d3ad37daa04 not found
{"provingJobId":"1:BASE_PARITY:299f9d3ad37daa04"}
09:43:37 [09:43:37.628]
INFO:
prover-client:proving-broker Proving Broker started
09:43:37 [09:43:37.628]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8e85ecfa4568f09e epochNumber=1
{"provingJobId":"1:BASE_PARITY:8e85ecfa4568f09e"}
09:43:37 [09:43:37.630]
INFO:
prover-client:proving-broker Proving Broker started
09:43:37 [09:43:37.631]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:95ad0d7b416aed54 epochNumber=1
{"provingJobId":"1:BASE_PARITY:95ad0d7b416aed54"}
09:43:37 [09:43:37.755]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:95ad0d7b416aed54 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:95ad0d7b416aed54"}
09:43:37 [09:43:37.760]
INFO:
prover-client:proving-broker Proving Broker started
09:43:37 [09:43:37.761]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52bd7992420d11f9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:52bd7992420d11f9"}
09:43:37 [09:43:37.862]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:52bd7992420d11f9 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:52bd7992420d11f9"}
09:43:37 [09:43:37.870]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:232fc230de12a5be epochNumber=10
{"provingJobId":"1:BASE_PARITY:232fc230de12a5be"}
09:43:37 [09:43:37.888]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:37 [09:43:37.996]
INFO:
prover-client:proving-broker Proving Broker started
09:43:37 [09:43:37.996]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cae4269863c1dfd0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:cae4269863c1dfd0"}
09:43:38 [09:43:38.173]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:cae4269863c1dfd0 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:cae4269863c1dfd0"}
09:43:38 [09:43:38.199]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.200]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f63551cc31d6bcd6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f63551cc31d6bcd6"}
09:43:38 [09:43:38.201]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f63551cc31d6bcd6 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:f63551cc31d6bcd6"}
09:43:38 [09:43:38.202]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.203]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0442dde154076dba epochNumber=1
{"provingJobId":"1:BASE_PARITY:0442dde154076dba"}
09:43:38 [09:43:38.204]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:0442dde154076dba type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:0442dde154076dba"}
09:43:38 [09:43:38.204]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0442dde154076dba type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:0442dde154076dba"}
09:43:38 [09:43:38.205]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.206]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9113d756c9d0b76c epochNumber=1
{"provingJobId":"1:BASE_PARITY:9113d756c9d0b76c"}
09:43:38 [09:43:38.207]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9113d756c9d0b76c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9113d756c9d0b76c"}
09:43:38 [09:43:38.208]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.209]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0b4b6606a222522 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e0b4b6606a222522"}
09:43:38 [09:43:38.209]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e0b4b6606a222522 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:e0b4b6606a222522"}
09:43:38 [09:43:38.210]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1940299f4ff0aa1f epochNumber=3
{"provingJobId":"1:BASE_PARITY:1940299f4ff0aa1f"}
09:43:38 [09:43:38.234]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:38 [09:43:38.235]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e0b4b6606a222522 err=test error
{"provingJoId":"1:BASE_PARITY:e0b4b6606a222522"}
09:43:38 [09:43:38.237]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.238]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:16a349307ea9076f settled=false
{"provingJobId":"1:BASE_PARITY:16a349307ea9076f","status":"pending"}
09:43:38 [09:43:38.238]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:3d32a21d71110c22 settled=false
{"provingJobId":"1:BASE_PARITY:3d32a21d71110c22","status":"pending"}
09:43:38 [09:43:38.240]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.240]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c00c122450e430ea settled=true
{"provingJobId":"1:BASE_PARITY:c00c122450e430ea","status":"fulfilled"}
09:43:38 [09:43:38.241]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:5f694072585387dc settled=true
{"provingJobId":"2:BASE_PARITY:5f694072585387dc","status":"fulfilled"}
09:43:38 [09:43:38.242]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.242]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:536a22f4b0ae71f3 settled=true
{"provingJobId":"1:BASE_PARITY:536a22f4b0ae71f3","status":"fulfilled"}
09:43:38 [09:43:38.242]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:23d9d5772b44cce6 settled=false
{"provingJobId":"1:BASE_PARITY:23d9d5772b44cce6","status":"pending"}
09:43:38 [09:43:38.243]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.245]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6624825f4d88d3f2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6624825f4d88d3f2"}
09:43:38 [09:43:38.247]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.247]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:219b8ad52ebfb4f3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:219b8ad52ebfb4f3"}
09:43:38 [09:43:38.248]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:219b8ad52ebfb4f3: Error: db error: Error: db error
09:43:38 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:219b8ad52ebfb4f3"}
09:43:38 [09:43:38.249]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.249]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e4ab895993e9e755 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e4ab895993e9e755"}
09:43:38 [09:43:38.250]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e4ab895993e9e755 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e4ab895993e9e755"}
09:43:38 [09:43:38.250]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e4ab895993e9e755 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e4ab895993e9e755"}
09:43:38 [09:43:38.251]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.251]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:57f2c4bc463f48af epochNumber=1
{"provingJobId":"1:BASE_PARITY:57f2c4bc463f48af"}
09:43:38 [09:43:38.251]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:57f2c4bc463f48af type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:57f2c4bc463f48af"}
09:43:38 [09:43:38.252]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:57f2c4bc463f48af type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:57f2c4bc463f48af"}
09:43:38 [09:43:38.252]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:57f2c4bc463f48af: Error: db error
09:43:38 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:57f2c4bc463f48af"}
09:43:38 [09:43:38.253]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.253]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:162be0eb4347df9c epochNumber=1
{"provingJobId":"1:BASE_PARITY:162be0eb4347df9c"}
09:43:38 [09:43:38.254]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:162be0eb4347df9c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:162be0eb4347df9c"}
09:43:38 [09:43:38.254]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:162be0eb4347df9c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:162be0eb4347df9c"}
09:43:38 [09:43:38.255]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ca10ddc4dc5028a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ca10ddc4dc5028a"}
09:43:38 [09:43:38.255]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6ca10ddc4dc5028a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:6ca10ddc4dc5028a"}
09:43:38 [09:43:38.255]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:6ca10ddc4dc5028a type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:6ca10ddc4dc5028a"}
09:43:38 [09:43:38.256]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:6ca10ddc4dc5028a jobErr=test error: Error: db error
09:43:38 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:6ca10ddc4dc5028a"}
09:43:38 [09:43:38.257]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.257]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c886e9b21882d7a8 not found
{"provingJobId":"1:BASE_PARITY:c886e9b21882d7a8"}
09:43:38 [09:43:38.258]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.259]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:60912bcff0ff1769 err=test error
{"provingJoId":"1:BASE_PARITY:60912bcff0ff1769"}
09:43:38 [09:43:38.259]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.270]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81c777d56755cfc0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:81c777d56755cfc0"}
09:43:38 [09:43:38.271]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:81c777d56755cfc0 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:81c777d56755cfc0"}
09:43:38 [09:43:38.271]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:81c777d56755cfc0 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:81c777d56755cfc0"}
09:43:38 [09:43:38.282]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:ba4eb1b464dd2488 epochNumber=2
{"provingJobId":"2:BASE_PARITY:ba4eb1b464dd2488"}
09:43:38 [09:43:38.282]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:ba4eb1b464dd2488 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:ba4eb1b464dd2488"}
09:43:38 [09:43:38.283]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:ba4eb1b464dd2488 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:ba4eb1b464dd2488"}
09:43:38 [09:43:38.345]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:65d87e758bfef6e7 epochNumber=3
{"provingJobId":"3:BASE_PARITY:65d87e758bfef6e7"}
09:43:38 [09:43:38.360]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:38 [09:43:38.406]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:a61923c1a134504f epochNumber=4
{"provingJobId":"4:BASE_PARITY:a61923c1a134504f"}
09:43:38 [09:43:38.412]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:38 [09:43:38.469]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:b6b18e712b6c162e epochNumber=5
{"provingJobId":"5:BASE_PARITY:b6b18e712b6c162e"}
09:43:38 [09:43:38.489]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:38 [09:43:38.525]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.526]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3a3ae5f73dfdad80 epochNumber=42
{"provingJobId":"1:BASE_PARITY:3a3ae5f73dfdad80"}
09:43:38 [09:43:38.526]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testXSkzmE/42 with map size 1073741824
09:43:38 [09:43:38.530]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.558]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:69e003ecf00e6774 epochNumber=1
{"provingJobId":"1:BASE_PARITY:69e003ecf00e6774"}
09:43:38 [09:43:38.562]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.563]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3f73136eb564f483 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3f73136eb564f483"}
09:43:38 [09:43:38.563]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHM0BCy/1 with map size 1073741824
09:43:38 [09:43:38.565]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.586]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02105d87c86c634b epochNumber=1
{"provingJobId":"1:BASE_PARITY:02105d87c86c634b"}
09:43:38 [09:43:38.595]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.595]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1a5f74126d5a288c epochNumber=1
{"provingJobId":"1:BASE_PARITY:1a5f74126d5a288c"}
09:43:38 [09:43:38.596]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testpSKN8l/1 with map size 1073741824
09:43:38 [09:43:38.598]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.615]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:1a5f74126d5a288c epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:1a5f74126d5a288c"}
09:43:38 [09:43:38.619]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.620]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd3ac29877f0b8c6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd3ac29877f0b8c6"}
09:43:38 [09:43:38.620]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testvysCOL/1 with map size 1073741824
09:43:38 [09:43:38.622]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.640]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fd3ac29877f0b8c6 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fd3ac29877f0b8c6"}
09:43:38 [09:43:38.641]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fd3ac29877f0b8c6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fd3ac29877f0b8c6"}
09:43:38 [09:43:38.647]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:fd3ac29877f0b8c6 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:fd3ac29877f0b8c6"}
09:43:38 [09:43:38.651]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.651]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c8a1a7ad99566c3e epochNumber=1
{"provingJobId":"1:BASE_PARITY:c8a1a7ad99566c3e"}
09:43:38 [09:43:38.652]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testjIfC5n/1 with map size 1073741824
09:43:38 [09:43:38.655]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.684]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.688]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.689]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2202ded28d68cba5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2202ded28d68cba5"}
09:43:38 [09:43:38.690]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsxPM8q/1 with map size 1073741824
09:43:38 [09:43:38.695]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.712]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2202ded28d68cba5
{"provingJobId":"1:BASE_PARITY:2202ded28d68cba5"}
09:43:38 [09:43:38.713]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2202ded28d68cba5 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:2202ded28d68cba5"}
09:43:38 [09:43:38.713]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2202ded28d68cba5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2202ded28d68cba5"}
09:43:38 [09:43:38.725]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.726]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14b0bb3aa4138ad7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:14b0bb3aa4138ad7"}
09:43:38 [09:43:38.726]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testecmq1e/1 with map size 1073741824
09:43:38 [09:43:38.730]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.751]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:14b0bb3aa4138ad7
{"provingJobId":"1:BASE_PARITY:14b0bb3aa4138ad7"}
09:43:38 [09:43:38.752]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:14b0bb3aa4138ad7 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:14b0bb3aa4138ad7"}
09:43:38 [09:43:38.764]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.765]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b29efbcb777d2c1f epochNumber=1
{"provingJobId":"1:BASE_PARITY:b29efbcb777d2c1f"}
09:43:38 [09:43:38.777]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test7zPElQ/1 with map size 1073741824
09:43:38 [09:43:38.781]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.802]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b29efbcb777d2c1f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b29efbcb777d2c1f"}
09:43:38 [09:43:38.803]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b29efbcb777d2c1f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b29efbcb777d2c1f"}
09:43:38 [09:43:38.811]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.812]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9f9655bce1f83f89 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9f9655bce1f83f89"}
09:43:38 [09:43:38.813]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testAGD7bW/1 with map size 1073741824
09:43:38 [09:43:38.816]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.834]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9f9655bce1f83f89 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:9f9655bce1f83f89"}
09:43:38 [09:43:38.835]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:9f9655bce1f83f89 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:9f9655bce1f83f89"}
09:43:38 [09:43:38.844]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.845]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:71fcac910c3886eb epochNumber=0
{"provingJobId":"1:BASE_PARITY:71fcac910c3886eb"}
09:43:38 [09:43:38.847]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testmHLgB3/0 with map size 1073741824
09:43:38 [09:43:38.849]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.879]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:71fcac910c3886eb epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:71fcac910c3886eb"}
09:43:38 [09:43:38.880]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6c06a65bf49fa1ee epochNumber=0
{"provingJobId":"1:BASE_PARITY:6c06a65bf49fa1ee"}
09:43:38 [09:43:38.880]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:71fcac910c3886eb epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:71fcac910c3886eb"}
09:43:38 [09:43:38.880]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d7775f4c52393c02 epochNumber=0
{"provingJobId":"1:BASE_PARITY:d7775f4c52393c02"}
09:43:38 [09:43:38.896]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.898]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.898]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c2bb6fadf2628a5d epochNumber=2
{"provingJobId":"1:BASE_PARITY:c2bb6fadf2628a5d"}
09:43:38 [09:43:38.899]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testBBcwv2/2 with map size 1073741824
09:43:38 [09:43:38.901]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.918]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54afc656adf9b6b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:54afc656adf9b6b8"}
09:43:38 [09:43:38.919]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testBBcwv2/1 with map size 1073741824
09:43:38 [09:43:38.921]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.939]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:315a368c65d6df13 epochNumber=1
{"provingJobId":"1:BASE_PARITY:315a368c65d6df13"}
09:43:38 [09:43:38.940]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testfTXypy/1 with map size 1073741824
09:43:38 [09:43:38.942]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.960]
INFO:
prover-client:proving-broker Proving Broker started
09:43:38 [09:43:38.961]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2864ed66b9ed4ba9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2864ed66b9ed4ba9"}
09:43:38 [09:43:38.961]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFmi4X8/1 with map size 1073741824
09:43:38 [09:43:38.963]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:38 [09:43:38.992]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dbaa083edce45557 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dbaa083edce45557"}
09:43:38 [09:43:38.998]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5ff9f204dd16ab8 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c5ff9f204dd16ab8"}
09:43:38 [09:43:38.999]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFmi4X8/2 with map size 1073741824
09:43:39 [09:43:39.001]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.018]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a23e8c55c4ded8a2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a23e8c55c4ded8a2"}
09:43:39 [09:43:39.028]
INFO:
prover-client:proving-broker Proving Broker started
09:43:39 [09:43:39.029]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0df456df37e914c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0df456df37e914c0"}
09:43:39 [09:43:39.029]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testlyhM93/1 with map size 1073741824
09:43:39 [09:43:39.031]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.047]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f04740b4835a1ce2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f04740b4835a1ce2"}
09:43:39 [09:43:39.053]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0a04df91c2f92d60 epochNumber=2
{"provingJobId":"1:BASE_PARITY:0a04df91c2f92d60"}
09:43:39 [09:43:39.054]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testlyhM93/2 with map size 1073741824
09:43:39 [09:43:39.055]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.077]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b05c067e2c02244e epochNumber=1
{"provingJobId":"1:BASE_PARITY:b05c067e2c02244e"}
09:43:39 [09:43:39.090]
INFO:
prover-client:proving-broker Proving Broker started
09:43:39 [09:43:39.091]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3d5a0a22d4f2e2a5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:3d5a0a22d4f2e2a5"}
09:43:39 [09:43:39.091]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testr2zSRs/1 with map size 1073741824
09:43:39 [09:43:39.093]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.108]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:695bb2f5ed29dbe7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:695bb2f5ed29dbe7"}
09:43:39 [09:43:39.114]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c52ce280003d1a95 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c52ce280003d1a95"}
09:43:39 [09:43:39.115]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testr2zSRs/2 with map size 1073741824
09:43:39 [09:43:39.117]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.131]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6e8846a1b4767d2e epochNumber=1
{"provingJobId":"1:BASE_PARITY:6e8846a1b4767d2e"}
09:43:39 [09:43:39.141]
INFO:
prover-client:proving-broker Proving Broker started
09:43:39 [09:43:39.142]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:65f110c000960715 epochNumber=1
{"provingJobId":"1:BASE_PARITY:65f110c000960715"}
09:43:39 [09:43:39.143]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5bUZ0G/1 with map size 1073741824
09:43:39 [09:43:39.145]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.162]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f4d89444369ad2a epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f4d89444369ad2a"}
09:43:39 [09:43:39.169]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:65f110c000960715 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:65f110c000960715"}
09:43:39 [09:43:39.185]
INFO:
prover-client:proving-broker Proving Broker started
09:43:39 [09:43:39.186]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5ca7e29cbfda4707 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5ca7e29cbfda4707"}
09:43:39 [09:43:39.187]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTWaE3j/1 with map size 1073741824
09:43:39 [09:43:39.190]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.209]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8ca38f7040e5b691 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8ca38f7040e5b691"}
09:43:39 [09:43:39.216]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5ca7e29cbfda4707 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:5ca7e29cbfda4707"}
09:43:39 [09:43:39.225]
INFO:
prover-client:proving-broker Proving Broker started
09:43:39 [09:43:39.226]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03787ddcf493b4ea epochNumber=1
{"provingJobId":"1:BASE_PARITY:03787ddcf493b4ea"}
09:43:39 [09:43:39.227]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testDHXmib/1 with map size 1073741824
09:43:39 [09:43:39.229]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.246]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:618f32373318086b epochNumber=1
{"provingJobId":"1:BASE_PARITY:618f32373318086b"}
09:43:39 [09:43:39.252]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:03787ddcf493b4ea type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:03787ddcf493b4ea"}
09:43:39 [09:43:39.255]
INFO:
prover-client:proving-broker Proving Broker started
09:43:39 [09:43:39.256]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f630322945eb548f epochNumber=1
{"provingJobId":"1:BASE_PARITY:f630322945eb548f"}
09:43:39 [09:43:39.256]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzRHULo/1 with map size 1073741824
09:43:39 [09:43:39.259]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.284]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f630322945eb548f
{"provingJobId":"1:BASE_PARITY:f630322945eb548f"}
09:43:39 [09:43:39.285]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f630322945eb548f type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f630322945eb548f"}
09:43:39 [09:43:39.291]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d869b4ea0a8bb059 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d869b4ea0a8bb059"}
09:43:39 [09:43:39.298]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f630322945eb548f has already been completed
{"provingJobId":"1:BASE_PARITY:f630322945eb548f"}
09:43:39 [09:43:39.302]
INFO:
prover-client:proving-broker Proving Broker started
09:43:39 [09:43:39.303]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee560991093ffc53 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee560991093ffc53"}
09:43:39 [09:43:39.304]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test2X61lu/1 with map size 1073741824
09:43:39 [09:43:39.306]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:39 [09:43:39.324]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c2e136b2b296fa5 epochNumber=2
{"provingJobId":"1:BASE_PARITY:2c2e136b2b296fa5"}
09:43:39 [09:43:39.325]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test2X61lu/2 with map size 1073741824
09:43:39 [09:43:39.328]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:40 [09:43:40.397]
INFO:
prover-client:proving-broker Proving Broker started
09:43:40 [09:43:40.404]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:ee560991093ffc53 settled=false
{"provingJobId":"1:BASE_PARITY:ee560991093ffc53","status":"pending"}
09:43:40 [09:43:40.406]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2c2e136b2b296fa5 settled=false
{"provingJobId":"1:BASE_PARITY:2c2e136b2b296fa5","status":"pending"}
09:43:40 [09:43:40.407]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ee560991093ffc53 type=BASE_PARITY startedAt=1750412619345 older agent has taken job
{"provingJobId":"1:BASE_PARITY:ee560991093ffc53"}
09:43:40 [09:43:40.407]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ee560991093ffc53 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:ee560991093ffc53"}
09:43:40 [09:43:40.411]
INFO:
prover-client:proving-broker Proving Broker started
09:43:40 [09:43:40.412]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:23490117d28f7669 epochNumber=1
{"provingJobId":"1:BASE_PARITY:23490117d28f7669"}
09:43:40 [09:43:40.413]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTaQASW/1 with map size 1073741824
09:43:40 [09:43:40.415]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:40 [09:43:40.438]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f4d01aa09a5b8537 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f4d01aa09a5b8537"}
09:43:40 [09:43:40.440]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testTaQASW/2 with map size 1073741824
09:43:40 [09:43:40.442]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:41 [09:43:41.461]
INFO:
prover-client:proving-broker Proving Broker started
09:43:41 [09:43:41.463]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:23490117d28f7669 settled=false
{"provingJobId":"1:BASE_PARITY:23490117d28f7669","status":"pending"}
09:43:41 [09:43:41.465]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:f4d01aa09a5b8537 settled=false
{"provingJobId":"1:BASE_PARITY:f4d01aa09a5b8537","status":"pending"}
09:43:41 [09:43:41.466]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:23490117d28f7669 type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:23490117d28f7669"}
09:43:41 [09:43:41.470]
INFO:
prover-client:proving-broker Proving Broker started
09:43:41 [09:43:41.471]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fd235f5326ae85a6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fd235f5326ae85a6"}
09:43:41 [09:43:41.471]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testb7gXGD/1 with map size 1073741824
09:43:41 [09:43:41.472]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:41 [09:43:41.493]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de7fd41444329da2 epochNumber=2
{"provingJobId":"1:BASE_PARITY:de7fd41444329da2"}
09:43:41 [09:43:41.494]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testb7gXGD/2 with map size 1073741824
09:43:41 [09:43:41.496]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:42 [09:43:42.510]
INFO:
prover-client:proving-broker Proving Broker started
09:43:42 [09:43:42.514]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:fd235f5326ae85a6 settled=false
{"provingJobId":"1:BASE_PARITY:fd235f5326ae85a6","status":"pending"}
09:43:42 [09:43:42.517]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:de7fd41444329da2 settled=false
{"provingJobId":"1:BASE_PARITY:de7fd41444329da2","status":"pending"}
09:43:42 [09:43:42.518]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:fd235f5326ae85a6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:fd235f5326ae85a6"}
09:43:42 [09:43:42.518]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:fd235f5326ae85a6 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:fd235f5326ae85a6"}
09:43:42 [09:43:42.527]
INFO:
prover-client:proving-broker Proving Broker started
09:43:42 [09:43:42.528]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2225a425accd63b1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2225a425accd63b1"}
09:43:42 [09:43:42.529]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZXFESG/1 with map size 1073741824
09:43:42 [09:43:42.532]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:42 [09:43:42.549]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1c518ea2c8d5be6b epochNumber=2
{"provingJobId":"1:BASE_PARITY:1c518ea2c8d5be6b"}
09:43:42 [09:43:42.550]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testZXFESG/2 with map size 1073741824
09:43:42 [09:43:42.551]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:42 [09:43:42.582]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:2225a425accd63b1 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:2225a425accd63b1"}
09:43:42 [09:43:42.590]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1c518ea2c8d5be6b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1c518ea2c8d5be6b"}
09:43:42 [09:43:42.599]
INFO:
prover-client:proving-broker Proving Broker started
09:43:42 [09:43:42.599]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:398757d36745ed65 epochNumber=1
{"provingJobId":"1:BASE_PARITY:398757d36745ed65"}
09:43:42 [09:43:42.600]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test3DC82o/1 with map size 1073741824
09:43:42 [09:43:42.604]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:42 [09:43:42.623]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f263f4ba09965e6b epochNumber=2
{"provingJobId":"1:BASE_PARITY:f263f4ba09965e6b"}
09:43:42 [09:43:42.624]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test3DC82o/2 with map size 1073741824
09:43:42 [09:43:42.628]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:42 [09:43:42.645]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:398757d36745ed65 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:398757d36745ed65"}
09:43:42 [09:43:42.646]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:398757d36745ed65 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:398757d36745ed65"}
09:43:42 [09:43:42.652]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f263f4ba09965e6b type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f263f4ba09965e6b"}
09:43:42 [09:43:42.661]
INFO:
prover-client:proving-broker Proving Broker started
09:43:42 [09:43:42.662]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:6b97ee45d2b7c95f err=test error
{"provingJoId":"1:BASE_PARITY:6b97ee45d2b7c95f"}
09:43:42 [09:43:42.686]
INFO:
prover-client:proving-broker Proving Broker started
09:43:42 [09:43:42.690]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:82ab1790fab2d573 not found
{"provingJobId":"1:BASE_PARITY:82ab1790fab2d573"}
09:43:42 [09:43:42.699]
INFO:
prover-client:proving-broker Proving Broker started
09:43:42 [09:43:42.700]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:74de4a5917fd705b epochNumber=1
{"provingJobId":"1:BASE_PARITY:74de4a5917fd705b"}
09:43:42 [09:43:42.701]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUt4o6m/1 with map size 1073741824
09:43:42 [09:43:42.705]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:42 [09:43:42.731]
INFO:
prover-client:proving-broker Proving Broker started
09:43:42 [09:43:42.732]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:386e14dc2306fe0a epochNumber=1
{"provingJobId":"1:BASE_PARITY:386e14dc2306fe0a"}
09:43:42 [09:43:42.734]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testMYo9U5/1 with map size 1073741824
09:43:42 [09:43:42.736]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:42 [09:43:42.858]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:386e14dc2306fe0a timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:386e14dc2306fe0a"}
09:43:42 [09:43:42.872]
INFO:
prover-client:proving-broker Proving Broker started
09:43:42 [09:43:42.873]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b448da763b0cae7d epochNumber=1
{"provingJobId":"1:BASE_PARITY:b448da763b0cae7d"}
09:43:42 [09:43:42.876]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testREIemr/1 with map size 1073741824
09:43:42 [09:43:42.885]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.026]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b448da763b0cae7d timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:b448da763b0cae7d"}
09:43:43 [09:43:43.031]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2f5bf0e8c3fb3b3e epochNumber=10
{"provingJobId":"1:BASE_PARITY:2f5bf0e8c3fb3b3e"}
09:43:43 [09:43:43.033]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testREIemr/10 with map size 1073741824
09:43:43 [09:43:43.036]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.080]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:43 [09:43:43.080]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
09:43:43 [09:43:43.188]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.189]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:875018aa98407a55 epochNumber=1
{"provingJobId":"1:BASE_PARITY:875018aa98407a55"}
09:43:43 [09:43:43.190]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testdmjfkd/1 with map size 1073741824
09:43:43 [09:43:43.193]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.368]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:875018aa98407a55 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:875018aa98407a55"}
09:43:43 [09:43:43.423]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.424]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d20c3be58ca58b13 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d20c3be58ca58b13"}
09:43:43 [09:43:43.424]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVIfuTd/1 with map size 1073741824
09:43:43 [09:43:43.426]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.444]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:d20c3be58ca58b13 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:d20c3be58ca58b13"}
09:43:43 [09:43:43.452]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.453]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eaf3e4f30038680d epochNumber=1
{"provingJobId":"1:BASE_PARITY:eaf3e4f30038680d"}
09:43:43 [09:43:43.454]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testZ2bPbS/1 with map size 1073741824
09:43:43 [09:43:43.456]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.479]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:eaf3e4f30038680d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:eaf3e4f30038680d"}
09:43:43 [09:43:43.480]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:eaf3e4f30038680d type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:eaf3e4f30038680d"}
09:43:43 [09:43:43.490]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.490]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1182c3c2e67cf893 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1182c3c2e67cf893"}
09:43:43 [09:43:43.491]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIYcAcN/1 with map size 1073741824
09:43:43 [09:43:43.493]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.509]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1182c3c2e67cf893 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1182c3c2e67cf893"}
09:43:43 [09:43:43.517]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.518]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:02d4a532f10bdbe9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:02d4a532f10bdbe9"}
09:43:43 [09:43:43.518]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testbaF7QK/1 with map size 1073741824
09:43:43 [09:43:43.520]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.535]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:02d4a532f10bdbe9 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:02d4a532f10bdbe9"}
09:43:43 [09:43:43.537]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c4f45bb9c215b854 epochNumber=3
{"provingJobId":"1:BASE_PARITY:c4f45bb9c215b854"}
09:43:43 [09:43:43.538]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testbaF7QK/3 with map size 1073741824
09:43:43 [09:43:43.540]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.568]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:43 [09:43:43.569]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
09:43:43 [09:43:43.579]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:02d4a532f10bdbe9 err=test error
{"provingJoId":"1:BASE_PARITY:02d4a532f10bdbe9"}
09:43:43 [09:43:43.588]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxurwEH/1 with map size 1073741824
09:43:43 [09:43:43.590]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.607]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testxurwEH/2 with map size 1073741824
09:43:43 [09:43:43.609]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.630]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.632]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:da3a7d485c1783c0 settled=false
{"provingJobId":"1:BASE_PARITY:da3a7d485c1783c0","status":"pending"}
09:43:43 [09:43:43.633]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:21192ef447936c12 settled=false
{"provingJobId":"1:BASE_PARITY:21192ef447936c12","status":"pending"}
09:43:43 [09:43:43.638]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test80h34L/1 with map size 1073741824
09:43:43 [09:43:43.641]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.665]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test80h34L/2 with map size 1073741824
09:43:43 [09:43:43.668]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.698]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.702]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2236c7f56820f5ea settled=true
{"provingJobId":"1:BASE_PARITY:2236c7f56820f5ea","status":"fulfilled"}
09:43:43 [09:43:43.704]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:932bcf1aeef03a52 settled=true
{"provingJobId":"2:BASE_PARITY:932bcf1aeef03a52","status":"fulfilled"}
09:43:43 [09:43:43.708]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testwruipe/1 with map size 1073741824
09:43:43 [09:43:43.713]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.736]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testwruipe/2 with map size 1073741824
09:43:43 [09:43:43.739]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.754]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.756]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:959ace6c75e0fd2c settled=true
{"provingJobId":"1:BASE_PARITY:959ace6c75e0fd2c","status":"fulfilled"}
09:43:43 [09:43:43.758]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:abe30de6cc44078f settled=false
{"provingJobId":"1:BASE_PARITY:abe30de6cc44078f","status":"pending"}
09:43:43 [09:43:43.765]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.767]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:db9be06295dab531 epochNumber=1
{"provingJobId":"1:BASE_PARITY:db9be06295dab531"}
09:43:43 [09:43:43.768]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrjCqS4/1 with map size 1073741824
09:43:43 [09:43:43.774]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.801]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.802]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5bfd0fa1cbb59171 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5bfd0fa1cbb59171"}
09:43:43 [09:43:43.803]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:5bfd0fa1cbb59171: Error: db error: Error: db error
09:43:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:5bfd0fa1cbb59171"}
09:43:43 [09:43:43.807]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.807]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e18af2e9b1c3649d epochNumber=1
{"provingJobId":"1:BASE_PARITY:e18af2e9b1c3649d"}
09:43:43 [09:43:43.808]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testOqkKII/1 with map size 1073741824
09:43:43 [09:43:43.809]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.831]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:e18af2e9b1c3649d type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:e18af2e9b1c3649d"}
09:43:43 [09:43:43.831]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:e18af2e9b1c3649d type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:e18af2e9b1c3649d"}
09:43:43 [09:43:43.841]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.842]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52810aaeed366767 epochNumber=1
{"provingJobId":"1:BASE_PARITY:52810aaeed366767"}
09:43:43 [09:43:43.843]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testq5dSPe/1 with map size 1073741824
09:43:43 [09:43:43.846]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.873]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:52810aaeed366767 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:52810aaeed366767"}
09:43:43 [09:43:43.874]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:52810aaeed366767 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:52810aaeed366767"}
09:43:43 [09:43:43.875]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:52810aaeed366767: Error: db error
09:43:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:52810aaeed366767"}
09:43:43 [09:43:43.880]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.881]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:825358ab28f1fbeb epochNumber=1
{"provingJobId":"1:BASE_PARITY:825358ab28f1fbeb"}
09:43:43 [09:43:43.883]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqZBxhR/1 with map size 1073741824
09:43:43 [09:43:43.887]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.908]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:825358ab28f1fbeb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:825358ab28f1fbeb"}
09:43:43 [09:43:43.909]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:825358ab28f1fbeb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:825358ab28f1fbeb"}
09:43:43 [09:43:43.924]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.925]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a6197068fe53574f epochNumber=1
{"provingJobId":"1:BASE_PARITY:a6197068fe53574f"}
09:43:43 [09:43:43.926]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testi4y9Sh/1 with map size 1073741824
09:43:43 [09:43:43.929]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:43 [09:43:43.947]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a6197068fe53574f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a6197068fe53574f"}
09:43:43 [09:43:43.948]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a6197068fe53574f type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a6197068fe53574f"}
09:43:43 [09:43:43.949]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a6197068fe53574f jobErr=test error: Error: db error
09:43:43 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:a6197068fe53574f"}
09:43:43 [09:43:43.956]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.957]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:3809994f36ca2f69 not found
{"provingJobId":"1:BASE_PARITY:3809994f36ca2f69"}
09:43:43 [09:43:43.961]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.961]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e6977e429122c822 err=test error
{"provingJoId":"1:BASE_PARITY:e6977e429122c822"}
09:43:43 [09:43:43.964]
INFO:
prover-client:proving-broker Proving Broker started
09:43:43 [09:43:43.975]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:34155595cebcbadc epochNumber=1
{"provingJobId":"1:BASE_PARITY:34155595cebcbadc"}
09:43:43 [09:43:43.976]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVI9aFX/1 with map size 1073741824
09:43:43 [09:43:43.980]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:44 [09:43:44.001]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:34155595cebcbadc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:34155595cebcbadc"}
09:43:44 [09:43:44.002]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:34155595cebcbadc type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:34155595cebcbadc"}
09:43:44 [09:43:44.019]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:8714f6e765c73552 epochNumber=2
{"provingJobId":"2:BASE_PARITY:8714f6e765c73552"}
09:43:44 [09:43:44.021]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVI9aFX/2 with map size 1073741824
09:43:44 [09:43:44.025]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:44 [09:43:44.045]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:8714f6e765c73552 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:8714f6e765c73552"}
09:43:44 [09:43:44.046]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:8714f6e765c73552 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:8714f6e765c73552"}
09:43:44 [09:43:44.112]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:ab13391e8d5fe8d3 epochNumber=3
{"provingJobId":"3:BASE_PARITY:ab13391e8d5fe8d3"}
09:43:44 [09:43:44.113]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testVI9aFX/3 with map size 1073741824
09:43:44 [09:43:44.117]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:44 [09:43:44.144]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:44 [09:43:44.144]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
09:43:44 [09:43:44.197]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:6d60210d89cd796e epochNumber=4
{"provingJobId":"4:BASE_PARITY:6d60210d89cd796e"}
09:43:44 [09:43:44.198]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testVI9aFX/4 with map size 1073741824
09:43:44 [09:43:44.200]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:44 [09:43:44.223]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:44 [09:43:44.223]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
09:43:44 [09:43:44.280]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:0b7f4233cb7ec091 epochNumber=5
{"provingJobId":"5:BASE_PARITY:0b7f4233cb7ec091"}
09:43:44 [09:43:44.281]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testVI9aFX/5 with map size 1073741824
09:43:44 [09:43:44.290]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
09:43:44 [09:43:44.326]
INFO:
prover-client:proving-broker Cleaned up jobs=1
09:43:44 [09:43:44.327]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
09:43:44
PASS src/proving_broker/proving_broker.test.ts (
13.056 s)
09:43:44 ProvingBroker
09:43:44 Producer API
09:43:44
✓ refuses stale jobs (38 ms)
09:43:44
✓ enqueues jobs (4 ms)
09:43:44
✓ ignores duplicate jobs (3 ms)
09:43:44
✓ reports correct status when enqueuing repeat jobs (3 ms)
09:43:44
✓ throws an error in case of duplicate job IDs (21 ms)
09:43:44
✓ returns not-found status for non-existing jobs (2 ms)
09:43:44
✓ cancels jobs in queue (3 ms)
09:43:44
✓ cancels jobs in-progress (2 ms)
09:43:44
✓ returns job result if successful (3 ms)
09:43:44
✓ returns job error if failed (3 ms)
09:43:44
✓ correctly returns job status for concurrent writes (11 ms)
09:43:44
✓ refuses stale jobs (39 ms)
09:43:44
✓ enqueues jobs (32 ms)
09:43:44
✓ ignores duplicate jobs (23 ms)
09:43:44
✓ reports correct status when enqueuing repeat jobs (30 ms)
09:43:44
✓ throws an error in case of duplicate job IDs (28 ms)
09:43:44
✓ returns not-found status for non-existing jobs (7 ms)
09:43:44
✓ cancels jobs in queue (38 ms)
09:43:44
✓ cancels jobs in-progress (37 ms)
09:43:44
✓ returns job result if successful (50 ms)
09:43:44
✓ returns job error if failed (32 ms)
09:43:44
✓ correctly returns job status for concurrent writes (51 ms)
09:43:44 Consumer API
09:43:44
✓ returns undefined if no jobs are available (3 ms)
09:43:44
✓ returns jobs in priority order (3 ms)
09:43:44
✓ returns undefined if no jobs are available for the given allowList (1 ms)
09:43:44
✓ returns a job if it is in the allowList (2 ms)
09:43:44
✓ returns the most important job if it is in the allowList (2 ms)
09:43:44
✓ returns any job if filter is empty (3 ms)
09:43:44
✓ returns a new job when reporting job success (4 ms)
09:43:44
✓ returns a new job when reporting permanent error (2 ms)
09:43:44
✓ returns a new job when reporting retry-able error (2 ms)
09:43:44
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
09:43:44
✓ returns a new job if job is already in progress elsewhere (1057 ms)
09:43:44
✓ avoids sending the same job to a new agent after a restart (1005 ms)
09:43:44
✓ avoids sending a completed job to a new agent after a restart (1009 ms)
09:43:44
✓ tracks job result if in progress (5 ms)
09:43:44
✓ tracks job result even if job is in queue (2 ms)
09:43:44
✓ ignores reported job error if unknown job (2 ms)
09:43:44
✓ ignores job result if unknown job (1 ms)
09:43:44
✓ returns undefined if no jobs are available (2 ms)
09:43:44
✓ returns jobs in priority order (40 ms)
09:43:44
✓ returns undefined if no jobs are available for the given allowList (21 ms)
09:43:44
✓ returns a job if it is in the allowList (67 ms)
09:43:44
✓ returns the most important job if it is in the allowList (61 ms)
09:43:44
✓ returns any job if filter is empty (50 ms)
09:43:44
✓ returns a new job when reporting job success (37 ms)
09:43:44
✓ returns a new job when reporting permanent error (46 ms)
09:43:44
✓ returns a new job when reporting retry-able error (31 ms)
09:43:44
✓ returns a new job when reporting progress if current one is cancelled (45 ms)
09:43:44
✓ returns a new job if job is already in progress elsewhere (1109 ms)
09:43:44
✓ avoids sending the same job to a new agent after a restart (1059 ms)
09:43:44
✓ avoids sending a completed job to a new agent after a restart (1057 ms)
09:43:44
✓ tracks job result if in progress (71 ms)
09:43:44
✓ tracks job result even if job is in queue (62 ms)
09:43:44
✓ ignores reported job error if unknown job (17 ms)
09:43:44
✓ ignores job result if unknown job (14 ms)
09:43:44 Timeouts
09:43:44
✓ tracks in progress jobs (3 ms)
09:43:44
✓ re-enqueues jobs that time out (130 ms)
09:43:44
✓ cancel stale jobs that time out (235 ms)
09:43:44
✓ keeps the jobs in progress while it is alive (204 ms)
09:43:44
✓ tracks in progress jobs (34 ms)
09:43:44
✓ re-enqueues jobs that time out (133 ms)
09:43:44
✓ cancel stale jobs that time out (320 ms)
09:43:44
✓ keeps the jobs in progress while it is alive (236 ms)
09:43:44 Retries
09:43:44
✓ retries jobs (3 ms)
09:43:44
✓ retries up to a maximum number of times (3 ms)
09:43:44
✓ passing retry=false does not retry (3 ms)
09:43:44
✓ does not retry if job is stale (28 ms)
09:43:44
✓ retries jobs (28 ms)
09:43:44
✓ retries up to a maximum number of times (39 ms)
09:43:44
✓ passing retry=false does not retry (27 ms)
09:43:44
✓ does not retry if job is stale (66 ms)
09:43:44 Database management
09:43:44
✓ re-enqueues proof requests on start (3 ms)
09:43:44
✓ restores proof results on start (1 ms)
09:43:44
✓ only re-enqueues unfinished jobs (2 ms)
09:43:44
✓ saves job when enqueued (3 ms)
09:43:44
✓ does not retain job if database fails to save (3 ms)
09:43:44
✓ saves job result (1 ms)
09:43:44
✓ saves result even if database fails to save (1 ms)
09:43:44
✓ saves job error (1 ms)
09:43:44
✓ saves job error even if database fails to save (3 ms)
09:43:44
✓ does not save job result if job is unknown (1 ms)
09:43:44
✓ does not save job error if job is unknown (1 ms)
09:43:44
✓ cleans up old jobs periodically (263 ms)
09:43:44
✓ re-enqueues proof requests on start (53 ms)
09:43:44
✓ restores proof results on start (69 ms)
09:43:44
✓ only re-enqueues unfinished jobs (55 ms)
09:43:44
✓ saves job when enqueued (37 ms)
09:43:44
✓ does not retain job if database fails to save (6 ms)
09:43:44
✓ saves job result (33 ms)
09:43:44
✓ saves result even if database fails to save (39 ms)
09:43:44
✓ saves job error (42 ms)
09:43:44
✓ saves job error even if database fails to save (33 ms)
09:43:44
✓ does not save job result if job is unknown (5 ms)
09:43:44
✓ does not save job error if job is unknown (3 ms)
09:43:44
✓ cleans up old jobs periodically (396 ms)
09:43:44
09:43:44
Test Suites: 1 passed, 1 total
09:43:44
Tests: 96 passed, 96 total
09:43:44
Snapshots: 0 total
09:43:44
Time: 13.153 s
09:43:44
Ran all test suites matching prover-client/src/proving_broker/proving_broker.test.ts
.
09:43:44
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?