Parent Log:
http://ci.aztec-labs.com/cc7478688846fa30
Command: c118fea9c063e772 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/proving_broker.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/12d931e5ea9343eec403068931d3121e1d10e37e
Env: REF_NAME=gh-readonly-queue/next/pr-15042-254faae7d330e160b14c6eaaa099dc92d4a9cac7 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 16 12:34:46 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15042_amd64_x3-full
Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_b95296ef06707e5f_next
12:34:50 [12:34:50.828]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.834]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8fa35831080d5a78 epochNumber=42
{"provingJobId":"1:BASE_PARITY:8fa35831080d5a78"}
12:34:50 [12:34:50.837]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:efa1da18834538de epochNumber=1
{"provingJobId":"1:BASE_PARITY:efa1da18834538de"}
12:34:50 [12:34:50.860]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.861]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ed3831ee5b597d3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ed3831ee5b597d3"}
12:34:50 [12:34:50.862]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:963bc53509f87014 epochNumber=1
{"provingJobId":"1:BASE_PARITY:963bc53509f87014"}
12:34:50 [12:34:50.863]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.864]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:59c85bde5d6f7282 epochNumber=1
{"provingJobId":"1:BASE_PARITY:59c85bde5d6f7282"}
12:34:50 [12:34:50.866]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:59c85bde5d6f7282 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:59c85bde5d6f7282"}
12:34:50 [12:34:50.870]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.871]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:58c89d04667ea2cd epochNumber=1
{"provingJobId":"1:BASE_PARITY:58c89d04667ea2cd"}
12:34:50 [12:34:50.872]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:58c89d04667ea2cd epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:58c89d04667ea2cd"}
12:34:50 [12:34:50.873]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:58c89d04667ea2cd type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:58c89d04667ea2cd"}
12:34:50 [12:34:50.874]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:58c89d04667ea2cd epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:58c89d04667ea2cd"}
12:34:50 [12:34:50.875]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.876]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6a8f3dcfca66d3eb epochNumber=1
{"provingJobId":"1:BASE_PARITY:6a8f3dcfca66d3eb"}
12:34:50 [12:34:50.914]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.916]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.917]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f392cfe09dc44be3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:f392cfe09dc44be3"}
12:34:50 [12:34:50.918]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:f392cfe09dc44be3
{"provingJobId":"1:BASE_PARITY:f392cfe09dc44be3"}
12:34:50 [12:34:50.919]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:f392cfe09dc44be3 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:f392cfe09dc44be3"}
12:34:50 [12:34:50.919]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f392cfe09dc44be3 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:f392cfe09dc44be3"}
12:34:50 [12:34:50.920]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.921]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:001cd242ce747a6b epochNumber=1
{"provingJobId":"1:BASE_PARITY:001cd242ce747a6b"}
12:34:50 [12:34:50.922]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:001cd242ce747a6b
{"provingJobId":"1:BASE_PARITY:001cd242ce747a6b"}
12:34:50 [12:34:50.922]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:001cd242ce747a6b type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:001cd242ce747a6b"}
12:34:50 [12:34:50.923]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.924]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:20c16190b7270d0b epochNumber=1
{"provingJobId":"1:BASE_PARITY:20c16190b7270d0b"}
12:34:50 [12:34:50.925]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:20c16190b7270d0b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:20c16190b7270d0b"}
12:34:50 [12:34:50.925]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:20c16190b7270d0b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:20c16190b7270d0b"}
12:34:50 [12:34:50.927]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.927]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b63309413a3c00fa epochNumber=1
{"provingJobId":"1:BASE_PARITY:b63309413a3c00fa"}
12:34:50 [12:34:50.928]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:b63309413a3c00fa type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:b63309413a3c00fa"}
12:34:50 [12:34:50.928]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b63309413a3c00fa type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b63309413a3c00fa"}
12:34:50 [12:34:50.929]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.930]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b283123ee1a54213 epochNumber=0
{"provingJobId":"1:BASE_PARITY:b283123ee1a54213"}
12:34:50 [12:34:50.930]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b283123ee1a54213 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b283123ee1a54213"}
12:34:50 [12:34:50.930]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eaade8b5ebb5b893 epochNumber=0
{"provingJobId":"1:BASE_PARITY:eaade8b5ebb5b893"}
12:34:50 [12:34:50.930]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:b283123ee1a54213 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:b283123ee1a54213"}
12:34:50 [12:34:50.931]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0401b167b3f49554 epochNumber=0
{"provingJobId":"1:BASE_PARITY:0401b167b3f49554"}
12:34:50 [12:34:50.932]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.933]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.934]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b6895cfd77f9269e epochNumber=2
{"provingJobId":"1:BASE_PARITY:b6895cfd77f9269e"}
12:34:50 [12:34:50.934]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9fe42f3b80508ba9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9fe42f3b80508ba9"}
12:34:50 [12:34:50.936]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.937]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2862d62eaf1e77c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2862d62eaf1e77c0"}
12:34:50 [12:34:50.938]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b080e7efaa373552 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b080e7efaa373552"}
12:34:50 [12:34:50.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8762e378649d14e1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8762e378649d14e1"}
12:34:50 [12:34:50.939]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52bb9c52c762a859 epochNumber=2
{"provingJobId":"1:BASE_PARITY:52bb9c52c762a859"}
12:34:50 [12:34:50.940]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1bcca54e652f3cff epochNumber=1
{"provingJobId":"1:BASE_PARITY:1bcca54e652f3cff"}
12:34:50 [12:34:50.941]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.941]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:66d10b0e90051366 epochNumber=1
{"provingJobId":"1:BASE_PARITY:66d10b0e90051366"}
12:34:50 [12:34:50.941]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2ac8866c2e059b8c epochNumber=1
{"provingJobId":"1:BASE_PARITY:2ac8866c2e059b8c"}
12:34:50 [12:34:50.942]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:36e5b45750459cf1 epochNumber=2
{"provingJobId":"1:BASE_PARITY:36e5b45750459cf1"}
12:34:50 [12:34:50.942]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c71ecccfb7ff7e21 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c71ecccfb7ff7e21"}
12:34:50 [12:34:50.943]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.943]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1b5a10393cc3ded3 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1b5a10393cc3ded3"}
12:34:50 [12:34:50.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:35cb2399f93ce707 epochNumber=1
{"provingJobId":"1:BASE_PARITY:35cb2399f93ce707"}
12:34:50 [12:34:50.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e0c89a9c60fb1092 epochNumber=2
{"provingJobId":"1:BASE_PARITY:e0c89a9c60fb1092"}
12:34:50 [12:34:50.944]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfe5d72db0f2eb9f epochNumber=1
{"provingJobId":"1:BASE_PARITY:dfe5d72db0f2eb9f"}
12:34:50 [12:34:50.945]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.945]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b991acfef767e291 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b991acfef767e291"}
12:34:50 [12:34:50.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:cf1de05e3644447d epochNumber=1
{"provingJobId":"1:BASE_PARITY:cf1de05e3644447d"}
12:34:50 [12:34:50.946]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:b991acfef767e291 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:b991acfef767e291"}
12:34:50 [12:34:50.947]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.948]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d66d11aabc1340a4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d66d11aabc1340a4"}
12:34:50 [12:34:50.948]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dc0588611dc096c7 epochNumber=1
{"provingJobId":"1:BASE_PARITY:dc0588611dc096c7"}
12:34:50 [12:34:50.948]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d66d11aabc1340a4 type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:d66d11aabc1340a4"}
12:34:50 [12:34:50.949]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.950]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fc9f4bdf59aa2398 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fc9f4bdf59aa2398"}
12:34:50 [12:34:50.950]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:87c7bcc1445e7c22 epochNumber=1
{"provingJobId":"1:BASE_PARITY:87c7bcc1445e7c22"}
12:34:50 [12:34:50.950]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fc9f4bdf59aa2398 type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:fc9f4bdf59aa2398"}
12:34:50 [12:34:50.951]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.952]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2c00bab581f44e42 epochNumber=1
{"provingJobId":"1:BASE_PARITY:2c00bab581f44e42"}
12:34:50 [12:34:50.952]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:2c00bab581f44e42
{"provingJobId":"1:BASE_PARITY:2c00bab581f44e42"}
12:34:50 [12:34:50.952]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:2c00bab581f44e42 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:2c00bab581f44e42"}
12:34:50 [12:34:50.953]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:09eca45e4250894b epochNumber=1
{"provingJobId":"1:BASE_PARITY:09eca45e4250894b"}
12:34:50 [12:34:50.953]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:2c00bab581f44e42 has already been completed
{"provingJobId":"1:BASE_PARITY:2c00bab581f44e42"}
12:34:50 [12:34:50.955]
INFO:
prover-client:proving-broker Proving Broker started
12:34:50 [12:34:50.956]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9141810817d69089 epochNumber=1
{"provingJobId":"1:BASE_PARITY:9141810817d69089"}
12:34:50 [12:34:50.956]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:19dfb46e7aeefe65 epochNumber=2
{"provingJobId":"1:BASE_PARITY:19dfb46e7aeefe65"}
12:34:52 [12:34:52.009]
INFO:
prover-client:proving-broker Proving Broker started
12:34:52 [12:34:52.010]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:9141810817d69089 settled=false
{"provingJobId":"1:BASE_PARITY:9141810817d69089","status":"pending"}
12:34:52 [12:34:52.011]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:19dfb46e7aeefe65 settled=false
{"provingJobId":"1:BASE_PARITY:19dfb46e7aeefe65","status":"pending"}
12:34:52 [12:34:52.012]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9141810817d69089 type=BASE_PARITY startedAt=1750077290957 older agent has taken job
{"provingJobId":"1:BASE_PARITY:9141810817d69089"}
12:34:52 [12:34:52.013]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:9141810817d69089 type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:9141810817d69089"}
12:34:52 [12:34:52.014]
INFO:
prover-client:proving-broker Proving Broker started
12:34:52 [12:34:52.015]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7bc8ed4e6d1d2fad epochNumber=1
{"provingJobId":"1:BASE_PARITY:7bc8ed4e6d1d2fad"}
12:34:52 [12:34:52.015]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:38dd4429a91f64ff epochNumber=2
{"provingJobId":"1:BASE_PARITY:38dd4429a91f64ff"}
12:34:53 [12:34:53.016]
INFO:
prover-client:proving-broker Proving Broker started
12:34:53 [12:34:53.017]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:7bc8ed4e6d1d2fad settled=false
{"provingJobId":"1:BASE_PARITY:7bc8ed4e6d1d2fad","status":"pending"}
12:34:53 [12:34:53.017]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:38dd4429a91f64ff settled=false
{"provingJobId":"1:BASE_PARITY:38dd4429a91f64ff","status":"pending"}
12:34:53 [12:34:53.018]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7bc8ed4e6d1d2fad type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:7bc8ed4e6d1d2fad"}
12:34:53 [12:34:53.024]
INFO:
prover-client:proving-broker Proving Broker started
12:34:53 [12:34:53.025]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4db3f7f86fa38350 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4db3f7f86fa38350"}
12:34:53 [12:34:53.026]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:54065746d4ab9c8e epochNumber=2
{"provingJobId":"1:BASE_PARITY:54065746d4ab9c8e"}
12:34:54 [12:34:54.027]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.028]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:4db3f7f86fa38350 settled=false
{"provingJobId":"1:BASE_PARITY:4db3f7f86fa38350","status":"pending"}
12:34:54 [12:34:54.029]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:54065746d4ab9c8e settled=false
{"provingJobId":"1:BASE_PARITY:54065746d4ab9c8e","status":"pending"}
12:34:54 [12:34:54.029]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4db3f7f86fa38350 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:4db3f7f86fa38350"}
12:34:54 [12:34:54.030]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:4db3f7f86fa38350 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:4db3f7f86fa38350"}
12:34:54 [12:34:54.032]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d3f2b589f6ba4adb epochNumber=1
{"provingJobId":"1:BASE_PARITY:d3f2b589f6ba4adb"}
12:34:54 [12:34:54.033]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a14cdcc82e4d7970 epochNumber=2
{"provingJobId":"1:BASE_PARITY:a14cdcc82e4d7970"}
12:34:54 [12:34:54.034]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:d3f2b589f6ba4adb type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:d3f2b589f6ba4adb"}
12:34:54 [12:34:54.034]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a14cdcc82e4d7970 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a14cdcc82e4d7970"}
12:34:54 [12:34:54.036]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.036]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ebbe37290fef31ca epochNumber=1
{"provingJobId":"1:BASE_PARITY:ebbe37290fef31ca"}
12:34:54 [12:34:54.037]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:72a8365e57afa48c epochNumber=2
{"provingJobId":"1:BASE_PARITY:72a8365e57afa48c"}
12:34:54 [12:34:54.037]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ebbe37290fef31ca type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:ebbe37290fef31ca"}
12:34:54 [12:34:54.037]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:ebbe37290fef31ca type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:ebbe37290fef31ca"}
12:34:54 [12:34:54.038]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:72a8365e57afa48c type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:72a8365e57afa48c"}
12:34:54 [12:34:54.040]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.042]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:7031df0a686768aa err=test error
{"provingJoId":"1:BASE_PARITY:7031df0a686768aa"}
12:34:54 [12:34:54.045]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.047]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:016575686b7c8df0 not found
{"provingJobId":"1:BASE_PARITY:016575686b7c8df0"}
12:34:54 [12:34:54.048]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.049]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b35e0aa4103a7929 epochNumber=1
{"provingJobId":"1:BASE_PARITY:b35e0aa4103a7929"}
12:34:54 [12:34:54.051]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.051]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6ce79d05d3ac08cb epochNumber=1
{"provingJobId":"1:BASE_PARITY:6ce79d05d3ac08cb"}
12:34:54 [12:34:54.155]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6ce79d05d3ac08cb timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:6ce79d05d3ac08cb"}
12:34:54 [12:34:54.161]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.162]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4f278890069dc09f epochNumber=1
{"provingJobId":"1:BASE_PARITY:4f278890069dc09f"}
12:34:54 [12:34:54.263]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4f278890069dc09f timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4f278890069dc09f"}
12:34:54 [12:34:54.265]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7359f03344ce7616 epochNumber=10
{"provingJobId":"1:BASE_PARITY:7359f03344ce7616"}
12:34:54 [12:34:54.289]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:34:54 [12:34:54.392]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.394]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:4b678e8c3cd38fd1 epochNumber=1
{"provingJobId":"1:BASE_PARITY:4b678e8c3cd38fd1"}
12:34:54 [12:34:54.549]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:4b678e8c3cd38fd1 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:4b678e8c3cd38fd1"}
12:34:54 [12:34:54.601]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.602]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:608461b7f579ba6a epochNumber=1
{"provingJobId":"1:BASE_PARITY:608461b7f579ba6a"}
12:34:54 [12:34:54.603]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:608461b7f579ba6a type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:608461b7f579ba6a"}
12:34:54 [12:34:54.605]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.605]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fa8db6376f839b34 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fa8db6376f839b34"}
12:34:54 [12:34:54.606]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:fa8db6376f839b34 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:fa8db6376f839b34"}
12:34:54 [12:34:54.607]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:fa8db6376f839b34 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:fa8db6376f839b34"}
12:34:54 [12:34:54.609]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.610]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a30ab1b1a5862e11 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a30ab1b1a5862e11"}
12:34:54 [12:34:54.611]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a30ab1b1a5862e11 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a30ab1b1a5862e11"}
12:34:54 [12:34:54.612]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.613]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:91794ba9990a6411 epochNumber=1
{"provingJobId":"1:BASE_PARITY:91794ba9990a6411"}
12:34:54 [12:34:54.614]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:91794ba9990a6411 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:91794ba9990a6411"}
12:34:54 [12:34:54.615]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12a173197d764356 epochNumber=3
{"provingJobId":"1:BASE_PARITY:12a173197d764356"}
12:34:54 [12:34:54.637]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:34:54 [12:34:54.640]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:91794ba9990a6411 err=test error
{"provingJoId":"1:BASE_PARITY:91794ba9990a6411"}
12:34:54 [12:34:54.642]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.642]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:38412ae69248daaa settled=false
{"provingJobId":"1:BASE_PARITY:38412ae69248daaa","status":"pending"}
12:34:54 [12:34:54.643]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:419b5ac311c938eb settled=false
{"provingJobId":"1:BASE_PARITY:419b5ac311c938eb","status":"pending"}
12:34:54 [12:34:54.645]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.645]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:75ea475fc86e250b settled=true
{"provingJobId":"1:BASE_PARITY:75ea475fc86e250b","status":"fulfilled"}
12:34:54 [12:34:54.646]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:3b40145c11f472f4 settled=true
{"provingJobId":"2:BASE_PARITY:3b40145c11f472f4","status":"fulfilled"}
12:34:54 [12:34:54.647]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.648]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:a29b18661e5612ac settled=true
{"provingJobId":"1:BASE_PARITY:a29b18661e5612ac","status":"fulfilled"}
12:34:54 [12:34:54.648]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:52809bccb26055ec settled=false
{"provingJobId":"1:BASE_PARITY:52809bccb26055ec","status":"pending"}
12:34:54 [12:34:54.652]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.653]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:250d525d7b1ba9bc epochNumber=1
{"provingJobId":"1:BASE_PARITY:250d525d7b1ba9bc"}
12:34:54 [12:34:54.656]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.657]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:131cb735307ec5b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:131cb735307ec5b4"}
12:34:54 [12:34:54.658]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:131cb735307ec5b4: Error: db error: Error: db error
12:34:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:131cb735307ec5b4"}
12:34:54 [12:34:54.659]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.660]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:76d83993e8ac4a7f epochNumber=1
{"provingJobId":"1:BASE_PARITY:76d83993e8ac4a7f"}
12:34:54 [12:34:54.660]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:76d83993e8ac4a7f type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:76d83993e8ac4a7f"}
12:34:54 [12:34:54.660]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:76d83993e8ac4a7f type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:76d83993e8ac4a7f"}
12:34:54 [12:34:54.663]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.666]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:39931eb410b16ab9 epochNumber=1
{"provingJobId":"1:BASE_PARITY:39931eb410b16ab9"}
12:34:54 [12:34:54.666]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:39931eb410b16ab9 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:39931eb410b16ab9"}
12:34:54 [12:34:54.667]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:39931eb410b16ab9 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:39931eb410b16ab9"}
12:34:54 [12:34:54.667]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:39931eb410b16ab9: Error: db error
12:34:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:39931eb410b16ab9"}
12:34:54 [12:34:54.669]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.670]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0038beb96d4fe8f6 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0038beb96d4fe8f6"}
12:34:54 [12:34:54.670]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:0038beb96d4fe8f6 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:0038beb96d4fe8f6"}
12:34:54 [12:34:54.670]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:0038beb96d4fe8f6 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:0038beb96d4fe8f6"}
12:34:54 [12:34:54.672]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.673]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a3021cdd48f825fb epochNumber=1
{"provingJobId":"1:BASE_PARITY:a3021cdd48f825fb"}
12:34:54 [12:34:54.673]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a3021cdd48f825fb type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:a3021cdd48f825fb"}
12:34:54 [12:34:54.673]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a3021cdd48f825fb type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:a3021cdd48f825fb"}
12:34:54 [12:34:54.675]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:a3021cdd48f825fb jobErr=test error: Error: db error
12:34:54 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:a3021cdd48f825fb"}
12:34:54 [12:34:54.677]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.678]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:304a35be0f2bb899 not found
{"provingJobId":"1:BASE_PARITY:304a35be0f2bb899"}
12:34:54 [12:34:54.680]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.681]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:9639f361984f8a95 err=test error
{"provingJoId":"1:BASE_PARITY:9639f361984f8a95"}
12:34:54 [12:34:54.682]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.693]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:16531bd84f3ae9ad epochNumber=1
{"provingJobId":"1:BASE_PARITY:16531bd84f3ae9ad"}
12:34:54 [12:34:54.694]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:16531bd84f3ae9ad type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:16531bd84f3ae9ad"}
12:34:54 [12:34:54.694]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:16531bd84f3ae9ad type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:16531bd84f3ae9ad"}
12:34:54 [12:34:54.704]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:4b5569e7c77ba043 epochNumber=2
{"provingJobId":"2:BASE_PARITY:4b5569e7c77ba043"}
12:34:54 [12:34:54.705]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:4b5569e7c77ba043 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:4b5569e7c77ba043"}
12:34:54 [12:34:54.706]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:4b5569e7c77ba043 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:4b5569e7c77ba043"}
12:34:54 [12:34:54.770]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:e9b8d6848615830b epochNumber=3
{"provingJobId":"3:BASE_PARITY:e9b8d6848615830b"}
12:34:54 [12:34:54.785]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:34:54 [12:34:54.833]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:84dc4541b6d85c80 epochNumber=4
{"provingJobId":"4:BASE_PARITY:84dc4541b6d85c80"}
12:34:54 [12:34:54.837]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:34:54 [12:34:54.896]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:d975be589a91d01d epochNumber=5
{"provingJobId":"5:BASE_PARITY:d975be589a91d01d"}
12:34:54 [12:34:54.913]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:34:54 [12:34:54.952]
INFO:
prover-client:proving-broker Proving Broker started
12:34:54 [12:34:54.953]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:52961d97e97f6cfe epochNumber=42
{"provingJobId":"1:BASE_PARITY:52961d97e97f6cfe"}
12:34:54 [12:34:54.955]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 42 at /tmp/proving-broker-testyko9n6/42 with map size 1073741824
12:34:54 [12:34:54.960]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:54 [12:34:54.996]
WARN:
prover-client:proving-broker Tried enqueueing stale proving job id=1:BASE_PARITY:e5f16bff33c8d031 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e5f16bff33c8d031"}
12:34:55 [12:34:55.005]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.006]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fad8f84211edbde2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fad8f84211edbde2"}
12:34:55 [12:34:55.008]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6DTpM7/1 with map size 1073741824
12:34:55 [12:34:55.013]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.035]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:81eca864297f7630 epochNumber=1
{"provingJobId":"1:BASE_PARITY:81eca864297f7630"}
12:34:55 [12:34:55.050]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.051]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:97124f7d7c5f8961 epochNumber=1
{"provingJobId":"1:BASE_PARITY:97124f7d7c5f8961"}
12:34:55 [12:34:55.052]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYyC0bO/1 with map size 1073741824
12:34:55 [12:34:55.055]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.076]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:97124f7d7c5f8961 epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:97124f7d7c5f8961"}
12:34:55 [12:34:55.081]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.082]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3b21bda37f1f288c epochNumber=1
{"provingJobId":"1:BASE_PARITY:3b21bda37f1f288c"}
12:34:55 [12:34:55.083]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt9ZbuP/1 with map size 1073741824
12:34:55 [12:34:55.088]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.114]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3b21bda37f1f288c epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3b21bda37f1f288c"}
12:34:55 [12:34:55.116]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:3b21bda37f1f288c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:3b21bda37f1f288c"}
12:34:55 [12:34:55.124]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:3b21bda37f1f288c epochNumber=1. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:3b21bda37f1f288c"}
12:34:55 [12:34:55.131]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.133]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9bd67f01d68363de epochNumber=1
{"provingJobId":"1:BASE_PARITY:9bd67f01d68363de"}
12:34:55 [12:34:55.133]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testqP4uZE/1 with map size 1073741824
12:34:55 [12:34:55.140]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.175]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.178]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.179]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5690d96ae648b918 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5690d96ae648b918"}
12:34:55 [12:34:55.180]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testJPLyp8/1 with map size 1073741824
12:34:55 [12:34:55.182]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.201]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:5690d96ae648b918
{"provingJobId":"1:BASE_PARITY:5690d96ae648b918"}
12:34:55 [12:34:55.202]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5690d96ae648b918 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5690d96ae648b918"}
12:34:55 [12:34:55.202]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:5690d96ae648b918 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:5690d96ae648b918"}
12:34:55 [12:34:55.213]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.214]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:eb7afd014e85c6ac epochNumber=1
{"provingJobId":"1:BASE_PARITY:eb7afd014e85c6ac"}
12:34:55 [12:34:55.214]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testkcTffz/1 with map size 1073741824
12:34:55 [12:34:55.218]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.243]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:eb7afd014e85c6ac
{"provingJobId":"1:BASE_PARITY:eb7afd014e85c6ac"}
12:34:55 [12:34:55.244]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:eb7afd014e85c6ac type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:eb7afd014e85c6ac"}
12:34:55 [12:34:55.255]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.255]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55e6aa4b4470e74c epochNumber=1
{"provingJobId":"1:BASE_PARITY:55e6aa4b4470e74c"}
12:34:55 [12:34:55.257]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testt0fHQd/1 with map size 1073741824
12:34:55 [12:34:55.264]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.281]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:55e6aa4b4470e74c type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:55e6aa4b4470e74c"}
12:34:55 [12:34:55.282]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:55e6aa4b4470e74c type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:55e6aa4b4470e74c"}
12:34:55 [12:34:55.297]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.298]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:207631963fec0872 epochNumber=1
{"provingJobId":"1:BASE_PARITY:207631963fec0872"}
12:34:55 [12:34:55.299]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testR8Sgfe/1 with map size 1073741824
12:34:55 [12:34:55.302]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.322]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:207631963fec0872 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:207631963fec0872"}
12:34:55 [12:34:55.323]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:207631963fec0872 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:207631963fec0872"}
12:34:55 [12:34:55.333]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.334]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9bd2585122bdab96 epochNumber=0
{"provingJobId":"1:BASE_PARITY:9bd2585122bdab96"}
12:34:55 [12:34:55.335]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 0 at /tmp/proving-broker-testaCcg7L/0 with map size 1073741824
12:34:55 [12:34:55.337]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.354]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9bd2585122bdab96 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9bd2585122bdab96"}
12:34:55 [12:34:55.355]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5f90bfc0a6a4cbfb epochNumber=0
{"provingJobId":"1:BASE_PARITY:5f90bfc0a6a4cbfb"}
12:34:55 [12:34:55.356]
WARN:
prover-client:proving-broker Cached proving job id=1:BASE_PARITY:9bd2585122bdab96 epochNumber=0. Not enqueuing again
{"provingJobId":"1:BASE_PARITY:9bd2585122bdab96"}
12:34:55 [12:34:55.356]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f2838a2186a35e53 epochNumber=0
{"provingJobId":"1:BASE_PARITY:f2838a2186a35e53"}
12:34:55 [12:34:55.371]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.373]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.374]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ddab1afe5e1f6a42 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ddab1afe5e1f6a42"}
12:34:55 [12:34:55.375]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test6OWiGm/2 with map size 1073741824
12:34:55 [12:34:55.376]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.391]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:03896dc9c158ddaa epochNumber=1
{"provingJobId":"1:BASE_PARITY:03896dc9c158ddaa"}
12:34:55 [12:34:55.392]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test6OWiGm/1 with map size 1073741824
12:34:55 [12:34:55.394]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.412]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.413]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:dfa8ac3c34cb136b epochNumber=1
{"provingJobId":"1:BASE_PARITY:dfa8ac3c34cb136b"}
12:34:55 [12:34:55.414]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testYxQX8E/1 with map size 1073741824
12:34:55 [12:34:55.416]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.436]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.437]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:9e0cd6f78b5029fd epochNumber=1
{"provingJobId":"1:BASE_PARITY:9e0cd6f78b5029fd"}
12:34:55 [12:34:55.437]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testFLSD3u/1 with map size 1073741824
12:34:55 [12:34:55.439]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.455]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e75cf0e1934f78f epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e75cf0e1934f78f"}
12:34:55 [12:34:55.460]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:acefb8f6a4193566 epochNumber=2
{"provingJobId":"1:BASE_PARITY:acefb8f6a4193566"}
12:34:55 [12:34:55.461]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testFLSD3u/2 with map size 1073741824
12:34:55 [12:34:55.463]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.479]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e36b67dd1133e900 epochNumber=1
{"provingJobId":"1:BASE_PARITY:e36b67dd1133e900"}
12:34:55 [12:34:55.489]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.490]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:48330708f181d261 epochNumber=1
{"provingJobId":"1:BASE_PARITY:48330708f181d261"}
12:34:55 [12:34:55.490]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test0bOa9F/1 with map size 1073741824
12:34:55 [12:34:55.492]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.508]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8f91e547371b23f0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8f91e547371b23f0"}
12:34:55 [12:34:55.514]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee5f4926dc3f8777 epochNumber=2
{"provingJobId":"1:BASE_PARITY:ee5f4926dc3f8777"}
12:34:55 [12:34:55.515]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-test0bOa9F/2 with map size 1073741824
12:34:55 [12:34:55.517]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.533]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:de2bf5c4ef393769 epochNumber=1
{"provingJobId":"1:BASE_PARITY:de2bf5c4ef393769"}
12:34:55 [12:34:55.543]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.544]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:55886fe3d4777286 epochNumber=1
{"provingJobId":"1:BASE_PARITY:55886fe3d4777286"}
12:34:55 [12:34:55.545]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testIgBf8Y/1 with map size 1073741824
12:34:55 [12:34:55.547]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.564]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:ee5ad4940b31f556 epochNumber=1
{"provingJobId":"1:BASE_PARITY:ee5ad4940b31f556"}
12:34:55 [12:34:55.570]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:12c98213a8cff123 epochNumber=2
{"provingJobId":"1:BASE_PARITY:12c98213a8cff123"}
12:34:55 [12:34:55.571]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testIgBf8Y/2 with map size 1073741824
12:34:55 [12:34:55.573]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.589]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:0b37c8d53e4df5b8 epochNumber=1
{"provingJobId":"1:BASE_PARITY:0b37c8d53e4df5b8"}
12:34:55 [12:34:55.599]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.600]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:94494637ea92ea39 epochNumber=1
{"provingJobId":"1:BASE_PARITY:94494637ea92ea39"}
12:34:55 [12:34:55.601]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testB69cCz/1 with map size 1073741824
12:34:55 [12:34:55.602]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.622]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:748ce724032889c0 epochNumber=1
{"provingJobId":"1:BASE_PARITY:748ce724032889c0"}
12:34:55 [12:34:55.628]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:94494637ea92ea39 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:94494637ea92ea39"}
12:34:55 [12:34:55.638]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.639]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d478f117c2e376ee epochNumber=1
{"provingJobId":"1:BASE_PARITY:d478f117c2e376ee"}
12:34:55 [12:34:55.639]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testu1NVzn/1 with map size 1073741824
12:34:55 [12:34:55.641]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.659]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7e0ec6cd5526e06f epochNumber=1
{"provingJobId":"1:BASE_PARITY:7e0ec6cd5526e06f"}
12:34:55 [12:34:55.664]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:d478f117c2e376ee type=BASE_PARITY totalAttempts=1 err=result
{"provingJobId":"1:BASE_PARITY:d478f117c2e376ee"}
12:34:55 [12:34:55.673]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.674]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f70263538de41fca epochNumber=1
{"provingJobId":"1:BASE_PARITY:f70263538de41fca"}
12:34:55 [12:34:55.675]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testli6D4F/1 with map size 1073741824
12:34:55 [12:34:55.677]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.706]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f629b8d49b95547c epochNumber=1
{"provingJobId":"1:BASE_PARITY:f629b8d49b95547c"}
12:34:55 [12:34:55.717]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:f70263538de41fca type=BASE_PARITY retry=1 err=result
{"provingJobId":"1:BASE_PARITY:f70263538de41fca"}
12:34:55 [12:34:55.738]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.739]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:7bc4b8bd257277d5 epochNumber=1
{"provingJobId":"1:BASE_PARITY:7bc4b8bd257277d5"}
12:34:55 [12:34:55.739]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test5xzVU3/1 with map size 1073741824
12:34:55 [12:34:55.741]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.764]
INFO:
prover-client:proving-broker Cancelling job id=1:BASE_PARITY:7bc4b8bd257277d5
{"provingJobId":"1:BASE_PARITY:7bc4b8bd257277d5"}
12:34:55 [12:34:55.765]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:7bc4b8bd257277d5 type=BASE_PARITY totalAttempts=1 err=Aborted
{"provingJobId":"1:BASE_PARITY:7bc4b8bd257277d5"}
12:34:55 [12:34:55.772]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:d4e7371be8bd79b2 epochNumber=1
{"provingJobId":"1:BASE_PARITY:d4e7371be8bd79b2"}
12:34:55 [12:34:55.778]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:7bc4b8bd257277d5 has already been completed
{"provingJobId":"1:BASE_PARITY:7bc4b8bd257277d5"}
12:34:55 [12:34:55.781]
INFO:
prover-client:proving-broker Proving Broker started
12:34:55 [12:34:55.782]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:14c5c8bd33c9a1ab epochNumber=1
{"provingJobId":"1:BASE_PARITY:14c5c8bd33c9a1ab"}
12:34:55 [12:34:55.783]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testmlYEoQ/1 with map size 1073741824
12:34:55 [12:34:55.786]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:55 [12:34:55.813]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c5e831cfb356b54b epochNumber=2
{"provingJobId":"1:BASE_PARITY:c5e831cfb356b54b"}
12:34:55 [12:34:55.820]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testmlYEoQ/2 with map size 1073741824
12:34:55 [12:34:55.832]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:56 [12:34:56.906]
INFO:
prover-client:proving-broker Proving Broker started
12:34:56 [12:34:56.927]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:14c5c8bd33c9a1ab settled=false
{"provingJobId":"1:BASE_PARITY:14c5c8bd33c9a1ab","status":"pending"}
12:34:56 [12:34:56.931]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c5e831cfb356b54b settled=false
{"provingJobId":"1:BASE_PARITY:c5e831cfb356b54b","status":"pending"}
12:34:56 [12:34:56.934]
INFO:
prover-client:proving-broker Proving job id=1:BASE_PARITY:14c5c8bd33c9a1ab type=BASE_PARITY startedAt=1750077295850 older agent has taken job
{"provingJobId":"1:BASE_PARITY:14c5c8bd33c9a1ab"}
12:34:56 [12:34:56.935]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:14c5c8bd33c9a1ab type=BASE_PARITY already being worked on by another agent. Sending new one
{"provingJobId":"1:BASE_PARITY:14c5c8bd33c9a1ab"}
12:34:56 [12:34:56.945]
INFO:
prover-client:proving-broker Proving Broker started
12:34:56 [12:34:56.946]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:6d17b482469da24a epochNumber=1
{"provingJobId":"1:BASE_PARITY:6d17b482469da24a"}
12:34:56 [12:34:56.948]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testsZLVCG/1 with map size 1073741824
12:34:56 [12:34:56.951]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:56 [12:34:56.973]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c04eba368a3425e0 epochNumber=2
{"provingJobId":"1:BASE_PARITY:c04eba368a3425e0"}
12:34:56 [12:34:56.974]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testsZLVCG/2 with map size 1073741824
12:34:56 [12:34:56.977]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:58 [12:34:58.011]
INFO:
prover-client:proving-broker Proving Broker started
12:34:58 [12:34:58.013]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:6d17b482469da24a settled=false
{"provingJobId":"1:BASE_PARITY:6d17b482469da24a","status":"pending"}
12:34:58 [12:34:58.014]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c04eba368a3425e0 settled=false
{"provingJobId":"1:BASE_PARITY:c04eba368a3425e0","status":"pending"}
12:34:58 [12:34:58.015]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:6d17b482469da24a type=BASE_PARITY not found in the in-progress cache, adding it
{"provingJobId":"1:BASE_PARITY:6d17b482469da24a"}
12:34:58 [12:34:58.019]
INFO:
prover-client:proving-broker Proving Broker started
12:34:58 [12:34:58.019]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5580eec2c812e939 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5580eec2c812e939"}
12:34:58 [12:34:58.020]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testrL8aqo/1 with map size 1073741824
12:34:58 [12:34:58.021]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:58 [12:34:58.039]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:2d8deafc95392cfa epochNumber=2
{"provingJobId":"1:BASE_PARITY:2d8deafc95392cfa"}
12:34:58 [12:34:58.040]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testrL8aqo/2 with map size 1073741824
12:34:58 [12:34:58.042]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.059]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.064]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:5580eec2c812e939 settled=false
{"provingJobId":"1:BASE_PARITY:5580eec2c812e939","status":"pending"}
12:34:59 [12:34:59.067]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:2d8deafc95392cfa settled=false
{"provingJobId":"1:BASE_PARITY:2d8deafc95392cfa","status":"pending"}
12:34:59 [12:34:59.068]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:5580eec2c812e939 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:5580eec2c812e939"}
12:34:59 [12:34:59.069]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:5580eec2c812e939 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:5580eec2c812e939"}
12:34:59 [12:34:59.079]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.080]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:07e804cfd24c184a epochNumber=1
{"provingJobId":"1:BASE_PARITY:07e804cfd24c184a"}
12:34:59 [12:34:59.080]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testU6V3xK/1 with map size 1073741824
12:34:59 [12:34:59.082]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.128]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:f59294628322b194 epochNumber=2
{"provingJobId":"1:BASE_PARITY:f59294628322b194"}
12:34:59 [12:34:59.132]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testU6V3xK/2 with map size 1073741824
12:34:59 [12:34:59.134]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.151]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:07e804cfd24c184a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:07e804cfd24c184a"}
12:34:59 [12:34:59.156]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:f59294628322b194 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:f59294628322b194"}
12:34:59 [12:34:59.166]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.166]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8c17678932679195 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8c17678932679195"}
12:34:59 [12:34:59.168]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testNyzPTc/1 with map size 1073741824
12:34:59 [12:34:59.170]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.186]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b84382a8eac71c90 epochNumber=2
{"provingJobId":"1:BASE_PARITY:b84382a8eac71c90"}
12:34:59 [12:34:59.187]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testNyzPTc/2 with map size 1073741824
12:34:59 [12:34:59.189]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.213]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:8c17678932679195 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:8c17678932679195"}
12:34:59 [12:34:59.214]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:8c17678932679195 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:8c17678932679195"}
12:34:59 [12:34:59.225]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:b84382a8eac71c90 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:b84382a8eac71c90"}
12:34:59 [12:34:59.236]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.237]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:af6b7ec55870d1b2 err=test error
{"provingJoId":"1:BASE_PARITY:af6b7ec55870d1b2"}
12:34:59 [12:34:59.239]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.240]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:ee2ba9c60fe81af2 not found
{"provingJobId":"1:BASE_PARITY:ee2ba9c60fe81af2"}
12:34:59 [12:34:59.244]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.244]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:053233ef77fd4281 epochNumber=1
{"provingJobId":"1:BASE_PARITY:053233ef77fd4281"}
12:34:59 [12:34:59.245]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testHipoC9/1 with map size 1073741824
12:34:59 [12:34:59.249]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.270]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.271]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:06cdc4714e9e5082 epochNumber=1
{"provingJobId":"1:BASE_PARITY:06cdc4714e9e5082"}
12:34:59 [12:34:59.271]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzGW31y/1 with map size 1073741824
12:34:59 [12:34:59.274]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.399]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:06cdc4714e9e5082 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:06cdc4714e9e5082"}
12:34:59 [12:34:59.422]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.424]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c7ba0db688a77fd4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:c7ba0db688a77fd4"}
12:34:59 [12:34:59.428]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testUmGMxV/1 with map size 1073741824
12:34:59 [12:34:59.431]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.552]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:c7ba0db688a77fd4 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:c7ba0db688a77fd4"}
12:34:59 [12:34:59.557]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:c259d9f83467d412 epochNumber=10
{"provingJobId":"1:BASE_PARITY:c259d9f83467d412"}
12:34:59 [12:34:59.558]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 10 at /tmp/proving-broker-testUmGMxV/10 with map size 1073741824
12:34:59 [12:34:59.561]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.604]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:34:59 [12:34:59.611]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:34:59 [12:34:59.721]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.723]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a4334ab76300f085 epochNumber=1
{"provingJobId":"1:BASE_PARITY:a4334ab76300f085"}
12:34:59 [12:34:59.727]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testTHvg5u/1 with map size 1073741824
12:34:59 [12:34:59.729]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.927]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a4334ab76300f085 timed out. Adding it back to the queue.
{"provingJobId":"1:BASE_PARITY:a4334ab76300f085"}
12:34:59 [12:34:59.961]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.962]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:13fed140e2363b7f epochNumber=1
{"provingJobId":"1:BASE_PARITY:13fed140e2363b7f"}
12:34:59 [12:34:59.963]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9HtCj1/1 with map size 1073741824
12:34:59 [12:34:59.965]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:34:59 [12:34:59.989]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:13fed140e2363b7f type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:13fed140e2363b7f"}
12:34:59 [12:34:59.993]
INFO:
prover-client:proving-broker Proving Broker started
12:34:59 [12:34:59.994]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:390c0c10f0546112 epochNumber=1
{"provingJobId":"1:BASE_PARITY:390c0c10f0546112"}
12:34:59 [12:34:59.995]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testf8uDbc/1 with map size 1073741824
12:35:00 [12:35:00.000]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.032]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:390c0c10f0546112 type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:390c0c10f0546112"}
12:35:00 [12:35:00.033]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:390c0c10f0546112 type=BASE_PARITY totalAttempts=2 err=test error
{"provingJobId":"1:BASE_PARITY:390c0c10f0546112"}
12:35:00 [12:35:00.042]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.043]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:8419d96b03312e64 epochNumber=1
{"provingJobId":"1:BASE_PARITY:8419d96b03312e64"}
12:35:00 [12:35:00.045]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-test9QEmBH/1 with map size 1073741824
12:35:00 [12:35:00.046]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.065]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:8419d96b03312e64 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:8419d96b03312e64"}
12:35:00 [12:35:00.078]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.079]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:e6528394eb29d86d epochNumber=1
{"provingJobId":"1:BASE_PARITY:e6528394eb29d86d"}
12:35:00 [12:35:00.080]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testSkvZqy/1 with map size 1073741824
12:35:00 [12:35:00.083]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.114]
INFO:
prover-client:proving-broker Retrying proving job id=1:BASE_PARITY:e6528394eb29d86d type=BASE_PARITY retry=1 err=test error
{"provingJobId":"1:BASE_PARITY:e6528394eb29d86d"}
12:35:00 [12:35:00.120]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:a856af9d3500a263 epochNumber=3
{"provingJobId":"1:BASE_PARITY:a856af9d3500a263"}
12:35:00 [12:35:00.126]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testSkvZqy/3 with map size 1073741824
12:35:00 [12:35:00.130]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.155]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:35:00 [12:35:00.155]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:35:00 [12:35:00.173]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:e6528394eb29d86d err=test error
{"provingJoId":"1:BASE_PARITY:e6528394eb29d86d"}
12:35:00 [12:35:00.179]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testzcBExP/1 with map size 1073741824
12:35:00 [12:35:00.185]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.233]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testzcBExP/2 with map size 1073741824
12:35:00 [12:35:00.236]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.255]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.258]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:0bad4ed949976e9a settled=false
{"provingJobId":"1:BASE_PARITY:0bad4ed949976e9a","status":"pending"}
12:35:00 [12:35:00.263]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:c6f549826c7d98f7 settled=false
{"provingJobId":"1:BASE_PARITY:c6f549826c7d98f7","status":"pending"}
12:35:00 [12:35:00.270]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testj3O8xo/1 with map size 1073741824
12:35:00 [12:35:00.275]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.298]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testj3O8xo/2 with map size 1073741824
12:35:00 [12:35:00.302]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.355]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.359]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:15ba291015c12e3d settled=true
{"provingJobId":"1:BASE_PARITY:15ba291015c12e3d","status":"fulfilled"}
12:35:00 [12:35:00.361]
INFO:
prover-client:proving-broker Restoring proving job id=2:BASE_PARITY:74f2288b5ed4bfea settled=true
{"provingJobId":"2:BASE_PARITY:74f2288b5ed4bfea","status":"fulfilled"}
12:35:00 [12:35:00.371]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testiSqa9r/1 with map size 1073741824
12:35:00 [12:35:00.373]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.397]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testiSqa9r/2 with map size 1073741824
12:35:00 [12:35:00.400]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.431]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.434]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:20a6df98db101820 settled=true
{"provingJobId":"1:BASE_PARITY:20a6df98db101820","status":"fulfilled"}
12:35:00 [12:35:00.436]
INFO:
prover-client:proving-broker Restoring proving job id=1:BASE_PARITY:d93f8f7c7e57e249 settled=false
{"provingJobId":"1:BASE_PARITY:d93f8f7c7e57e249","status":"pending"}
12:35:00 [12:35:00.443]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.444]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:5c4866dc2f817c49 epochNumber=1
{"provingJobId":"1:BASE_PARITY:5c4866dc2f817c49"}
12:35:00 [12:35:00.444]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testPrmM23/1 with map size 1073741824
12:35:00 [12:35:00.446]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.474]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.475]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:fabd0a92e77bd6b4 epochNumber=1
{"provingJobId":"1:BASE_PARITY:fabd0a92e77bd6b4"}
12:35:00 [12:35:00.476]
ERROR:
prover-client:proving-broker Failed to save proving job id=1:BASE_PARITY:fabd0a92e77bd6b4: Error: db error: Error: db error
12:35:00 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1126:63)
{"provingJobId":"1:BASE_PARITY:fabd0a92e77bd6b4"}
12:35:00 [12:35:00.480]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.480]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:32e3aa8734e89f4a epochNumber=1
{"provingJobId":"1:BASE_PARITY:32e3aa8734e89f4a"}
12:35:00 [12:35:00.481]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testX7KjYL/1 with map size 1073741824
12:35:00 [12:35:00.485]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.510]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:32e3aa8734e89f4a type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:32e3aa8734e89f4a"}
12:35:00 [12:35:00.510]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:32e3aa8734e89f4a type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:32e3aa8734e89f4a"}
12:35:00 [12:35:00.528]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.529]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:21b3f47dace7fb76 epochNumber=1
{"provingJobId":"1:BASE_PARITY:21b3f47dace7fb76"}
12:35:00 [12:35:00.532]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testhExn7Z/1 with map size 1073741824
12:35:00 [12:35:00.535]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.559]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:21b3f47dace7fb76 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:21b3f47dace7fb76"}
12:35:00 [12:35:00.560]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:21b3f47dace7fb76 type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:21b3f47dace7fb76"}
12:35:00 [12:35:00.561]
ERROR:
prover-client:proving-broker Failed to save proving job result id=1:BASE_PARITY:21b3f47dace7fb76: Error: db error
12:35:00 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1159:69)
{"provingJobId":"1:BASE_PARITY:21b3f47dace7fb76"}
12:35:00 [12:35:00.567]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.568]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:1e425de104279f75 epochNumber=1
{"provingJobId":"1:BASE_PARITY:1e425de104279f75"}
12:35:00 [12:35:00.569]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testQeY7rw/1 with map size 1073741824
12:35:00 [12:35:00.572]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.589]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:1e425de104279f75 type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:1e425de104279f75"}
12:35:00 [12:35:00.590]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:1e425de104279f75 type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:1e425de104279f75"}
12:35:00 [12:35:00.602]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.603]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:716c24a4cb3172dc epochNumber=1
{"provingJobId":"1:BASE_PARITY:716c24a4cb3172dc"}
12:35:00 [12:35:00.605]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testxqrAGb/1 with map size 1073741824
12:35:00 [12:35:00.611]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.635]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:716c24a4cb3172dc type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:716c24a4cb3172dc"}
12:35:00 [12:35:00.636]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:716c24a4cb3172dc type=BASE_PARITY totalAttempts=1 err=test error
{"provingJobId":"1:BASE_PARITY:716c24a4cb3172dc"}
12:35:00 [12:35:00.637]
ERROR:
prover-client:proving-broker Failed to save proving job error status id=1:BASE_PARITY:716c24a4cb3172dc jobErr=test error: Error: db error
12:35:00 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_broker.test.ts:1192:68)
{"provingJobId":"1:BASE_PARITY:716c24a4cb3172dc"}
12:35:00 [12:35:00.643]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.644]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:a303b14ebd23e4b4 not found
{"provingJobId":"1:BASE_PARITY:a303b14ebd23e4b4"}
12:35:00 [12:35:00.652]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.654]
WARN:
prover-client:proving-broker Can't set error on unknown proving job id=1:BASE_PARITY:9c9a56dea9568a7d err=test error
{"provingJoId":"1:BASE_PARITY:9c9a56dea9568a7d"}
12:35:00 [12:35:00.661]
INFO:
prover-client:proving-broker Proving Broker started
12:35:00 [12:35:00.672]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:aafa4bbb9782188b epochNumber=1
{"provingJobId":"1:BASE_PARITY:aafa4bbb9782188b"}
12:35:00 [12:35:00.674]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 1 at /tmp/proving-broker-testVj611L/1 with map size 1073741824
12:35:00 [12:35:00.676]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.693]
WARN:
prover-client:proving-broker Proving job id=1:BASE_PARITY:aafa4bbb9782188b type=BASE_PARITY not in the in-progress set
{"provingJobId":"1:BASE_PARITY:aafa4bbb9782188b"}
12:35:00 [12:35:00.693]
INFO:
prover-client:proving-broker Proving job complete id=1:BASE_PARITY:aafa4bbb9782188b type=BASE_PARITY totalAttempts=1
{"provingJobId":"1:BASE_PARITY:aafa4bbb9782188b"}
12:35:00 [12:35:00.710]
INFO:
prover-client:proving-broker New proving job id=2:BASE_PARITY:3a6242d8712ca018 epochNumber=2
{"provingJobId":"2:BASE_PARITY:3a6242d8712ca018"}
12:35:00 [12:35:00.712]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 2 at /tmp/proving-broker-testVj611L/2 with map size 1073741824
12:35:00 [12:35:00.717]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.734]
WARN:
prover-client:proving-broker Proving job id=2:BASE_PARITY:3a6242d8712ca018 type=BASE_PARITY not in the in-progress set
{"provingJobId":"2:BASE_PARITY:3a6242d8712ca018"}
12:35:00 [12:35:00.735]
INFO:
prover-client:proving-broker Proving job complete id=2:BASE_PARITY:3a6242d8712ca018 type=BASE_PARITY totalAttempts=1
{"provingJobId":"2:BASE_PARITY:3a6242d8712ca018"}
12:35:00 [12:35:00.804]
INFO:
prover-client:proving-broker New proving job id=3:BASE_PARITY:e9d22aae84917981 epochNumber=3
{"provingJobId":"3:BASE_PARITY:e9d22aae84917981"}
12:35:00 [12:35:00.805]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 3 at /tmp/proving-broker-testVj611L/3 with map size 1073741824
12:35:00 [12:35:00.810]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.840]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:35:00 [12:35:00.841]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 1
12:35:00 [12:35:00.894]
INFO:
prover-client:proving-broker New proving job id=4:BASE_PARITY:75977321679b427c epochNumber=4
{"provingJobId":"4:BASE_PARITY:75977321679b427c"}
12:35:00 [12:35:00.895]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 4 at /tmp/proving-broker-testVj611L/4 with map size 1073741824
12:35:00 [12:35:00.898]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:00 [12:35:00.921]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:35:00 [12:35:00.921]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 2
12:35:00 [12:35:00.979]
INFO:
prover-client:proving-broker New proving job id=5:BASE_PARITY:1f1c368b29ece765 epochNumber=5
{"provingJobId":"5:BASE_PARITY:1f1c368b29ece765"}
12:35:00 [12:35:00.980]
INFO:
prover-client:proving-broker-database Creating broker database for epoch 5 at /tmp/proving-broker-testVj611L/5 with map size 1073741824
12:35:00 [12:35:00.983]
INFO:
kv-store:lmdb-v2 Starting data store with maxReaders 16
12:35:01 [12:35:01.001]
INFO:
prover-client:proving-broker Cleaned up jobs=1
12:35:01 [12:35:01.001]
INFO:
prover-client:proving-broker-database Deleting broker database for epoch 3
12:35:01
PASS src/proving_broker/proving_broker.test.ts (
13.5 s)
12:35:01 ProvingBroker
12:35:01 Producer API
12:35:01
✓ refuses stale jobs (35 ms)
12:35:01
✓ enqueues jobs (3 ms)
12:35:01
✓ ignores duplicate jobs (6 ms)
12:35:01
✓ reports correct status when enqueuing repeat jobs (5 ms)
12:35:01
✓ throws an error in case of duplicate job IDs (38 ms)
12:35:01
✓ returns not-found status for non-existing jobs (1 ms)
12:35:01
✓ cancels jobs in queue (4 ms)
12:35:01
✓ cancels jobs in-progress (2 ms)
12:35:01
✓ returns job result if successful (3 ms)
12:35:01
✓ returns job error if failed (2 ms)
12:35:01
✓ correctly returns job status for concurrent writes (2 ms)
12:35:01
✓ refuses stale jobs (50 ms)
12:35:01
✓ enqueues jobs (46 ms)
12:35:01
✓ ignores duplicate jobs (31 ms)
12:35:01
✓ reports correct status when enqueuing repeat jobs (48 ms)
12:35:01
✓ throws an error in case of duplicate job IDs (44 ms)
12:35:01
✓ returns not-found status for non-existing jobs (4 ms)
12:35:01
✓ cancels jobs in queue (34 ms)
12:35:01
✓ cancels jobs in-progress (40 ms)
12:35:01
✓ returns job result if successful (40 ms)
12:35:01
✓ returns job error if failed (37 ms)
12:35:01
✓ correctly returns job status for concurrent writes (37 ms)
12:35:01 Consumer API
12:35:01
✓ returns undefined if no jobs are available (1 ms)
12:35:01
✓ returns jobs in priority order (2 ms)
12:35:01
✓ returns undefined if no jobs are available for the given allowList (1 ms)
12:35:01
✓ returns a job if it is in the allowList (2 ms)
12:35:01
✓ returns the most important job if it is in the allowList (2 ms)
12:35:01
✓ returns any job if filter is empty (2 ms)
12:35:01
✓ returns a new job when reporting job success (2 ms)
12:35:01
✓ returns a new job when reporting permanent error (2 ms)
12:35:01
✓ returns a new job when reporting retry-able error (2 ms)
12:35:01
✓ returns a new job when reporting progress if current one is cancelled (3 ms)
12:35:01
✓ returns a new job if job is already in progress elsewhere (1059 ms)
12:35:01
✓ avoids sending the same job to a new agent after a restart (1009 ms)
12:35:01
✓ avoids sending a completed job to a new agent after a restart (1008 ms)
12:35:01
✓ tracks job result if in progress (4 ms)
12:35:01
✓ tracks job result even if job is in queue (3 ms)
12:35:01
✓ ignores reported job error if unknown job (4 ms)
12:35:01
✓ ignores job result if unknown job (3 ms)
12:35:01
✓ returns undefined if no jobs are available (2 ms)
12:35:01
✓ returns jobs in priority order (39 ms)
12:35:01
✓ returns undefined if no jobs are available for the given allowList (20 ms)
12:35:01
✓ returns a job if it is in the allowList (55 ms)
12:35:01
✓ returns the most important job if it is in the allowList (53 ms)
12:35:01
✓ returns any job if filter is empty (55 ms)
12:35:01
✓ returns a new job when reporting job success (38 ms)
12:35:01
✓ returns a new job when reporting permanent error (35 ms)
12:35:01
✓ returns a new job when reporting retry-able error (63 ms)
12:35:01
✓ returns a new job when reporting progress if current one is cancelled (44 ms)
12:35:01
✓ returns a new job if job is already in progress elsewhere (1158 ms)
12:35:01
✓ avoids sending the same job to a new agent after a restart (1078 ms)
12:35:01
✓ avoids sending a completed job to a new agent after a restart (1059 ms)
12:35:01
✓ tracks job result if in progress (87 ms)
12:35:01
✓ tracks job result even if job is in queue (70 ms)
12:35:01
✓ ignores reported job error if unknown job (3 ms)
12:35:01
✓ ignores job result if unknown job (3 ms)
12:35:01 Timeouts
12:35:01
✓ tracks in progress jobs (2 ms)
12:35:01
✓ re-enqueues jobs that time out (110 ms)
12:35:01
✓ cancel stale jobs that time out (231 ms)
12:35:01
✓ keeps the jobs in progress while it is alive (207 ms)
12:35:01
✓ tracks in progress jobs (27 ms)
12:35:01
✓ re-enqueues jobs that time out (133 ms)
12:35:01
✓ cancel stale jobs that time out (306 ms)
12:35:01
✓ keeps the jobs in progress while it is alive (244 ms)
12:35:01 Retries
12:35:01
✓ retries jobs (4 ms)
12:35:01
✓ retries up to a maximum number of times (4 ms)
12:35:01
✓ passing retry=false does not retry (4 ms)
12:35:01
✓ does not retry if job is stale (28 ms)
12:35:01
✓ retries jobs (32 ms)
12:35:01
✓ retries up to a maximum number of times (49 ms)
12:35:01
✓ passing retry=false does not retry (32 ms)
12:35:01
✓ does not retry if job is stale (101 ms)
12:35:01 Database management
12:35:01
✓ re-enqueues proof requests on start (3 ms)
12:35:01
✓ restores proof results on start (2 ms)
12:35:01
✓ only re-enqueues unfinished jobs (4 ms)
12:35:01
✓ saves job when enqueued (4 ms)
12:35:01
✓ does not retain job if database fails to save (3 ms)
12:35:01
✓ saves job result (3 ms)
12:35:01
✓ saves result even if database fails to save (6 ms)
12:35:01
✓ saves job error (2 ms)
12:35:01
✓ saves job error even if database fails to save (4 ms)
12:35:01
✓ does not save job result if job is unknown (2 ms)
12:35:01
✓ does not save job error if job is unknown (2 ms)
12:35:01
✓ cleans up old jobs periodically (268 ms)
12:35:01
✓ re-enqueues proof requests on start (91 ms)
12:35:01
✓ restores proof results on start (98 ms)
12:35:01
✓ only re-enqueues unfinished jobs (75 ms)
12:35:01
✓ saves job when enqueued (31 ms)
12:35:01
✓ does not retain job if database fails to save (6 ms)
12:35:01
✓ saves job result (47 ms)
12:35:01
✓ saves result even if database fails to save (37 ms)
12:35:01
✓ saves job error (35 ms)
12:35:01
✓ saves job error even if database fails to save (40 ms)
12:35:01
✓ does not save job result if job is unknown (5 ms)
12:35:01
✓ does not save job error if job is unknown (10 ms)
12:35:01
✓ cleans up old jobs periodically (397 ms)
12:35:01
12:35:01
Test Suites: 1 passed, 1 total
12:35:01
Tests: 96 passed, 96 total
12:35:01
Snapshots: 0 total
12:35:01
Time: 13.589 s
12:35:01
Ran all test suites matching /prover-client\/src\/proving_broker\/proving_broker.test.ts/i
.
12:35:01
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?