Parent Log:
http://ci.aztec-labs.com/54ddd17842ec1f11
Command: d062a9b47b7331ac:ISOLATE=1:NAME=prover-client/src/test/bb_prover_full_rollup.test.ts:CPUS=16:MEM=96g LOG_LEVEL=verbose yarn-project/scripts/run_test.sh prover-client/src/test/bb_prover_full_rollup.test.ts
Commit:
https://github.com/AztecProtocol/aztec-packages/commit/6a9ca072a09e3c61d21e2d518f0fbf0275548eb3
Env: REF_NAME=gh-readonly-queue/next/pr-14858-96a426245731b3f48fcb6dd5e2a683c858706602 CURRENT_VERSION=0.87.6 CI_FULL=1
Date: Mon Jun 16 22:16:27 UTC 2025
System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14858_amd64_x2-full
Resources: CPU_LIST=0-127 CPUS=16 MEM=96g TIMEOUT=600s
History:
http://ci.aztec-labs.com/list/history_be20a80131a4ae4a_next
22:16:27 +++ id -u
22:16:27 +++ id -g
22:16:27 ++ docker run -d --name prover-client_src_test_bb_prover_full_rollup.test.ts --net=none --cpuset-cpus=0-127 --cpus=16 --memory=96g --user 1000:1000 -v/home/aztec-dev:/home/aztec-dev --mount type=tmpfs,target=/tmp,tmpfs-size=1g --workdir /home/aztec-dev/aztec-packages -e HOME -e VERBOSE -e GIT_CONFIG_GLOBAL=/home/aztec-dev/aztec-packages/build-images/src/home/.gitconfig -e FORCE_COLOR=true -e CPUS -e MEM aztecprotocol/build:3.0 /bin/bash -c 'timeout -v 600s bash -c '\''LOG_LEVEL=verbose yarn-project/scripts/run_test.sh prover-client/src/test/bb_prover_full_rollup.test.ts'\'''
22:16:28 + cid=debb91cb7f0f5c7ca3941f949bf2c56c2e4e0471fdb7ac5e7deaec9271ac279f
22:16:28 + set +x
22:16:30 {"level":25,"time":1750112190418,"pid":7,"hostname":"debb91cb7f0f","module":"logger","msg":"Logger initialized with level verbose"}
22:16:39 [22:16:39.794]
INFO:
world-state:database Creating world state data store at directory /tmp/aztec-world-state-COL0yl/world_state with map sizes {"archiveTreeMapSizeKb":10485760,"nullifierTreeMapSizeKb":10485760,"noteHashTreeMapSizeKb":10485760,"messageTreeMapSizeKb":10485760,"publicDataTreeMapSizeKb":10485760} and 16 threads.
22:16:39 [22:16:39.880]
INFO:
prover-client:test:bb-prover-full-rollup Found native BB binary at /home/aztec-dev/aztec-packages/barretenberg/cpp/build/bin/bb with working directory /tmp/f622934a/bb
22:16:39 [22:16:39.882]
INFO:
prover-client:test:bb-prover-full-rollup Found native ACVM binary at /home/aztec-dev/aztec-packages/noir/noir-repo/target/release/acvm with working directory /tmp/f622934a/acvm
22:16:39 [22:16:39.883]
INFO:
bb-prover Using native BB at /home/aztec-dev/aztec-packages/barretenberg/cpp/build/bin/bb and working directory /tmp/f622934a/bb
22:16:39 [22:16:39.884]
INFO:
bb-prover Using native ACVM at /home/aztec-dev/aztec-packages/noir/noir-repo/target/release/acvm and working directory /tmp/f622934a/acvm
22:16:39 [22:16:39.887]
INFO:
prover-client:proving-broker Proving Broker started
22:16:39 [22:16:39.889]
VERBOSE:
prover-client:broker-circuit-prover-facade Starting BrokerCircuitProverFacade
22:16:39 [22:16:39.891]
INFO:
prover-client:test:bb-prover-full-rollup Proving epoch with 1/1 blocks with 0/2 non-empty txs
22:16:39 [22:16:39.892]
INFO:
prover-client:test:bb-prover-full-rollup Processing public functions
22:16:39 [22:16:39.894]
INFO:
simulator:public-processor Processed 0 successful txs and 0 failed txs in 0.00006889200000114216s
{"duration":0.00006889200000114216,"rate":0,"totalPublicGas":{"daGas":0,"l2Gas":0},"totalBlockGas":{"daGas":0,"l2Gas":0},"totalSizeInBytes":0}
22:16:39 [22:16:39.911]
INFO:
prover-client:orchestrator Starting epoch 1 with 1 blocks
22:16:39 [22:16:39.914]
INFO:
prover-client:test:bb-prover-full-rollup Starting new block #1
22:16:39 [22:16:39.916]
INFO:
prover-client:orchestrator Starting block 1 for slot 1
22:16:39 [22:16:39.936]
WARN:
prover-client:orchestrator Provided no txs to orchestrator addTxs.
22:16:39 [22:16:39.940]
INFO:
prover-client:test:bb-prover-full-rollup Setting block as completed
22:16:39 [22:16:39.943]
VERBOSE:
prover-client:orchestrator Block 1 completed. Assembling header.
22:16:39 [22:16:39.949]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f epochNumber=1
{"provingJobId":"1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f"}
22:16:39 [22:16:39.950]
VERBOSE:
prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f type=BASE_PARITY epochNumber=1
{"provingJobId":"1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":1}
22:16:39 [22:16:39.953]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39 epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39"}
22:16:39 [22:16:39.953]
VERBOSE:
prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39 type=BASE_PARITY epochNumber=1
{"provingJobId":"1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":2}
22:16:39 [22:16:39.954]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:908c704315ae42e71259cec86f20559128254786cbedd665ce435727691b7d08 epochNumber=1
{"provingJobId":"1:BASE_PARITY:908c704315ae42e71259cec86f20559128254786cbedd665ce435727691b7d08"}
22:16:39 [22:16:39.955]
VERBOSE:
prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:908c704315ae42e71259cec86f20559128254786cbedd665ce435727691b7d08 type=BASE_PARITY epochNumber=1
{"provingJobId":"1:BASE_PARITY:908c704315ae42e71259cec86f20559128254786cbedd665ce435727691b7d08","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":3}
22:16:39 [22:16:39.956]
INFO:
prover-client:proving-broker New proving job id=1:BASE_PARITY:b5b09a6d9cdc01312ddd19f506884a71b67f905cfdd47e8baf08dbd1604263da epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5b09a6d9cdc01312ddd19f506884a71b67f905cfdd47e8baf08dbd1604263da"}
22:16:39 [22:16:39.964]
VERBOSE:
prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:b5b09a6d9cdc01312ddd19f506884a71b67f905cfdd47e8baf08dbd1604263da type=BASE_PARITY epochNumber=1
{"provingJobId":"1:BASE_PARITY:b5b09a6d9cdc01312ddd19f506884a71b67f905cfdd47e8baf08dbd1604263da","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":4}
22:16:39 [22:16:39.986]
VERBOSE:
prover-client:orchestrator Updating archive tree with block 1 header 0x2c7c24629329ca2756416dad177f07cd7ada0d48ac9b564496c3a1926779e766
22:16:39 [22:16:39.993]
INFO:
prover-client:proving-agent Starting job id=1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...
22:16:39 [22:16:39.994]
INFO:
prover-client:proving-agent:job-controller-1c31c509 Job controller started jobId=1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f
{"jobId":"1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f"}
22:16:39 [22:16:39.995]
VERBOSE:
prover-client:orchestrator Orchestrator finalised block 1
22:16:40 [22:16:40.009]
INFO:
prover-client:test:bb-prover-full-rollup Updating world state with new block
22:16:40 [22:16:40.109]
INFO:
prover-client:test:bb-prover-full-rollup Awaiting proofs
22:16:40 [22:16:40.206]
INFO:
bb-prover Generated witness
{"circuitName":"base-parity","duration":45.361383999999816,"inputSize":160,"outputSize":96,"eventName":"circuit-witness-generation"}
22:16:44 [22:16:44.861]
ERROR:
bb-prover Error running operation at /tmp/f622934a/bb/tmp-kQxfQa: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456
22:16:44 [22:16:44.874]
ERROR:
prover-client:proving-agent Job id=1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f type=BASE_PARITY failed err=Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 retry=false: Error: Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456
22:16:44 at ProvingJobController.run (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:101:18) {
22:16:44 [cause]: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456
22:16:44 at readProofAsFields (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/proof_utils.js:43:5)
22:16:44 at operation (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/server/bb_prover.js:315:27)
22:16:44 at runInDirectory (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/fs/run_in_dir.js:11:16)
22:16:44 at BBNativeRollupProver.createRecursiveProof (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/server/bb_prover.js:335:16)
22:16:44 at BBNativeRollupProver.getBaseParityProof (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/server/bb_prover.js:63:42)
22:16:44 at ProvingJobController.generateProof (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:164:16)
22:16:44 at ProvingJobController.run (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:96:16) {
22:16:44 generatedMessage: false,
22:16:44 code: 'ERR_ASSERTION',
22:16:44 actual: false,
22:16:44 expected: true,
22:16:44 operator: '=='
22:16:44 }
22:16:44 }
22:16:44 [22:16:44.875]
INFO:
prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f type=BASE_PARITY totalAttempts=1 err=Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456
{"provingJobId":"1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f"}
22:16:44 [22:16:44.876]
INFO:
prover-client:proving-agent Starting job id=1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39 type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...
22:16:44 [22:16:44.877]
INFO:
prover-client:proving-agent:job-controller-3123ab1b Job controller started jobId=1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39
{"jobId":"1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39"}
22:16:44 [22:16:44.905]
VERBOSE:
prover-client:broker-circuit-prover-facade Check for job completion notifications returned 1 job(s), snapshot ids length: 0, num outstanding jobs: 4, total jobs ready: 1
22:16:44 [22:16:44.906]
ERROR:
prover-client:broker-circuit-prover-facade Resolving proving job with error id=1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f type=BASE_PARITY: 'Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456'
22:16:44 [22:16:44.906]
ERROR:
prover-client:broker-circuit-prover-facade Job errored with 'Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456' id=1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f type=BASE_PARITY: {
22:16:44 provingJobId: '1:BASE_PARITY:3434d33b7e56d65b0fe6d665cf971261d295479447406f48a74d09fe714fee0f',
22:16:44 provingJobType: 'BASE_PARITY'
22:16:44 }
22:16:44 [22:16:44.907]
VERBOSE:
prover-client:broker-circuit-prover-facade Successfully retrieved 1 of 1 jobs that should be ready, total ready jobs is now: 0
22:16:44 [22:16:44.926]
ERROR:
prover-client:orchestrator Error thrown when proving job: Error: Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456
22:16:44 at processJob (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:328:29)
22:16:44 at async Promise.all (index 0)
22:16:44 at BrokerCircuitProverFacade.retrieveJobsThatShouldBeReady (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:348:23)
22:16:44 at BrokerCircuitProverFacade.monitorForCompletedJobs (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:388:5)
22:16:44 at poll (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/promise/running-promise.js:48:21)
22:16:44 [22:16:44.929]
VERBOSE:
prover-client:broker-circuit-prover-facade Stopping BrokerCircuitProverFacade
22:16:44 [22:16:44.930]
ERROR:
prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39 type=BASE_PARITY: {
22:16:44 provingJobId: '1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39',
22:16:44 provingJobType: 'BASE_PARITY'
22:16:44 }
22:16:44 [22:16:44.931]
ERROR:
prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=1:BASE_PARITY:908c704315ae42e71259cec86f20559128254786cbedd665ce435727691b7d08 type=BASE_PARITY: {
22:16:44 provingJobId: '1:BASE_PARITY:908c704315ae42e71259cec86f20559128254786cbedd665ce435727691b7d08',
22:16:44 provingJobType: 'BASE_PARITY'
22:16:44 }
22:16:44 [22:16:44.931]
ERROR:
prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=1:BASE_PARITY:b5b09a6d9cdc01312ddd19f506884a71b67f905cfdd47e8baf08dbd1604263da type=BASE_PARITY: {
22:16:44 provingJobId: '1:BASE_PARITY:b5b09a6d9cdc01312ddd19f506884a71b67f905cfdd47e8baf08dbd1604263da',
22:16:44 provingJobType: 'BASE_PARITY'
22:16:44 }
22:16:44 [22:16:44.933]
WARN:
prover-client:proving-agent:job-controller-3123ab1b Aborted job controller for jobId=1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39
{"jobId":"1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39"}
22:16:44 [22:16:44.943]
ERROR:
prover-client:orchestrator Error thrown when proving job: Error: Broker facade stopped
22:16:44 at BrokerCircuitProverFacade.stop (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:200:25)
22:16:44 at TestContext.cleanup (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/mocks/test_context.ts:150:5)
22:16:44 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts:39:5)
22:16:44 [22:16:44.944]
ERROR:
prover-client:orchestrator Error thrown when proving job: Error: Broker facade stopped
22:16:44 at BrokerCircuitProverFacade.stop (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:200:25)
22:16:44 at TestContext.cleanup (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/mocks/test_context.ts:150:5)
22:16:44 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts:39:5)
22:16:44 [22:16:44.945]
ERROR:
prover-client:orchestrator Error thrown when proving job: Error: Broker facade stopped
22:16:44 at BrokerCircuitProverFacade.stop (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:200:25)
22:16:44 at TestContext.cleanup (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/mocks/test_context.ts:150:5)
22:16:44 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts:39:5)
22:16:44 [22:16:44.953]
ERROR:
bb-prover Error running operation at /tmp/f622934a/bb/tmp-6thBNm: Error: Failed to generate witness: Error: ENOENT: no such file or directory, copyfile '/tmp/f622934a/acvm/tmp-0nJctG/output-witness.gz' -> '/tmp/f622934a/bb/tmp-6thBNm/partial-witness.gz'
22:16:44 [22:16:44.954]
WARN:
prover-client:proving-agent:job-controller-3123ab1b Job controller for jobId=1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39 completed but job was aborted
{"currentStatus":"running","jobId":"1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39"}
22:16:44 [22:16:44.955]
WARN:
prover-client:proving-agent Job id=1:BASE_PARITY:bd0bded81fbdbd612a073c064a5a6ce85a9c32528716a84d457c20228c822d39 was aborted. Not reporting result back to broker
22:16:44 err: {
22:16:44 "type": "AbortError",
22:16:44 "message": "Proof was aborted",
22:16:44 "stack":
22:16:44 AbortError: Proof was aborted
22:16:44 at ProvingJobController.run (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:110:16)
22:16:44 "name": "AbortError"
22:16:44 }
22:16:45
FAIL src/test/bb_prover_full_rollup.test.ts (
14.808 s)
22:16:45 prover/bb_prover/full-rollup
22:16:45
✕ proves a private-only epoch with 1/1 blocks with 0/2 non-empty txs each (5674 ms)
22:16:45
○ skipped proves all circuits
22:16:45
22:16:45
● prover/bb_prover/full-rollup › proves a private-only epoch with 1/1 blocks with 0/2 non-empty txs each
22:16:45
22:16:45 Epoch proving failed: Error: Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456
22:16:45
22:16:45
375 | const result
= await this.provingPromise
!;
22:16:45
376 | if (result.status === 'failure') {
22:16:45
> 377 | throw new Error(`Epoch proving failed: ${result.reason}`);
22:16:45
| ^
22:16:45
378 | }
22:16:45
379 |
22:16:45
380 | // TODO(MW): Move this? Requires async and don't want to force root methods to be async
22:16:45
22:16:45
at TestProvingOrchestrator.finaliseEpoch (orchestrator/orchestrator.ts:377:13)
22:16:45
at test/bb_prover_full_rollup.test.ts:94:27
22:16:45
22:16:45
Test Suites: 1 failed, 1 total
22:16:45
Tests: 1 failed,
1 skipped, 2 total
22:16:45
Snapshots: 0 total
22:16:45
Time: 14.921 s
22:16:45
Ran all test suites matching prover-client/src/test/bb_prover_full_rollup.test.ts
.
22:16:45
Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?