Parent Log: http://ci.aztec-labs.com/fefd49421fdd5679 Command: d062a9b47b7331ac:ISOLATE=1:NAME=prover-client/src/test/bb_prover_full_rollup.test.ts:CPUS=16:MEM=96g LOG_LEVEL=verbose yarn-project/scripts/run_test.sh prover-client/src/test/bb_prover_full_rollup.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/6a9ca072a09e3c61d21e2d518f0fbf0275548eb3 Env: REF_NAME=gh-readonly-queue/next/pr-14858-96a426245731b3f48fcb6dd5e2a683c858706602 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Mon Jun 16 22:16:29 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14858_amd64_x1-full Resources: CPU_LIST=0-127 CPUS=16 MEM=96g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_be20a80131a4ae4a_next 22:16:29 +++ id -u 22:16:29 +++ id -g 22:16:29 ++ docker run -d --name prover-client_src_test_bb_prover_full_rollup.test.ts --net=none --cpuset-cpus=0-127 --cpus=16 --memory=96g --user 1000:1000 -v/home/aztec-dev:/home/aztec-dev --mount type=tmpfs,target=/tmp,tmpfs-size=1g --workdir /home/aztec-dev/aztec-packages -e HOME -e VERBOSE -e GIT_CONFIG_GLOBAL=/home/aztec-dev/aztec-packages/build-images/src/home/.gitconfig -e FORCE_COLOR=true -e CPUS -e MEM aztecprotocol/build:3.0 /bin/bash -c 'timeout -v 600s bash -c '\''LOG_LEVEL=verbose yarn-project/scripts/run_test.sh prover-client/src/test/bb_prover_full_rollup.test.ts'\''' 22:16:30 + cid=aa4eb099311d0b219f89a21625ab6dac6b8012ce48ed3e03315a22d5321a9d8f 22:16:30 + set +x 22:16:31 {"level":25,"time":1750112191801,"pid":7,"hostname":"aa4eb099311d","module":"logger","msg":"Logger initialized with level verbose"} 22:16:41 [22:16:41.858] INFO: world-state:database Creating world state data store at directory /tmp/aztec-world-state-ywDwSb/world_state with map sizes {"archiveTreeMapSizeKb":10485760,"nullifierTreeMapSizeKb":10485760,"noteHashTreeMapSizeKb":10485760,"messageTreeMapSizeKb":10485760,"publicDataTreeMapSizeKb":10485760} and 16 threads. 22:16:41 [22:16:41.950] INFO: prover-client:test:bb-prover-full-rollup Found native BB binary at /home/aztec-dev/aztec-packages/barretenberg/cpp/build/bin/bb with working directory /tmp/b243a500/bb 22:16:41 [22:16:41.954] INFO: prover-client:test:bb-prover-full-rollup Found native ACVM binary at /home/aztec-dev/aztec-packages/noir/noir-repo/target/release/acvm with working directory /tmp/b243a500/acvm 22:16:41 [22:16:41.955] INFO: bb-prover Using native BB at /home/aztec-dev/aztec-packages/barretenberg/cpp/build/bin/bb and working directory /tmp/b243a500/bb 22:16:41 [22:16:41.956] INFO: bb-prover Using native ACVM at /home/aztec-dev/aztec-packages/noir/noir-repo/target/release/acvm and working directory /tmp/b243a500/acvm 22:16:41 [22:16:41.966] INFO: prover-client:proving-broker Proving Broker started 22:16:41 [22:16:41.968] VERBOSE: prover-client:broker-circuit-prover-facade Starting BrokerCircuitProverFacade 22:16:41 [22:16:41.971] INFO: prover-client:test:bb-prover-full-rollup Proving epoch with 1/1 blocks with 0/2 non-empty txs 22:16:41 [22:16:41.972] INFO: prover-client:test:bb-prover-full-rollup Processing public functions 22:16:41 [22:16:41.974] INFO: simulator:public-processor Processed 0 successful txs and 0 failed txs in 0.00008520599999974365s {"duration":0.00008520599999974365,"rate":0,"totalPublicGas":{"daGas":0,"l2Gas":0},"totalBlockGas":{"daGas":0,"l2Gas":0},"totalSizeInBytes":0} 22:16:41 [22:16:41.991] INFO: prover-client:orchestrator Starting epoch 1 with 1 blocks 22:16:41 [22:16:41.994] INFO: prover-client:test:bb-prover-full-rollup Starting new block #1 22:16:41 [22:16:41.995] INFO: prover-client:orchestrator Starting block 1 for slot 1 22:16:42 [22:16:42.023] WARN: prover-client:orchestrator Provided no txs to orchestrator addTxs. 22:16:42 [22:16:42.024] INFO: prover-client:test:bb-prover-full-rollup Setting block as completed 22:16:42 [22:16:42.026] VERBOSE: prover-client:orchestrator Block 1 completed. Assembling header. 22:16:42 [22:16:42.031] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c epochNumber=1 {"provingJobId":"1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c"} 22:16:42 [22:16:42.034] VERBOSE: prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c type=BASE_PARITY epochNumber=1 {"provingJobId":"1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":1} 22:16:42 [22:16:42.036] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d epochNumber=1 {"provingJobId":"1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d"} 22:16:42 [22:16:42.038] VERBOSE: prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d type=BASE_PARITY epochNumber=1 {"provingJobId":"1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":2} 22:16:42 [22:16:42.040] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:f7e3fa88037136d09cfd8f15ed0db469032e74ec740a5f87facb52fd6601669e epochNumber=1 {"provingJobId":"1:BASE_PARITY:f7e3fa88037136d09cfd8f15ed0db469032e74ec740a5f87facb52fd6601669e"} 22:16:42 [22:16:42.041] VERBOSE: prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:f7e3fa88037136d09cfd8f15ed0db469032e74ec740a5f87facb52fd6601669e type=BASE_PARITY epochNumber=1 {"provingJobId":"1:BASE_PARITY:f7e3fa88037136d09cfd8f15ed0db469032e74ec740a5f87facb52fd6601669e","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":3} 22:16:42 [22:16:42.042] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:83b97e35c6fcbb174b36cc01443732667146db1483c1011b6fa6834945e1aa76 epochNumber=1 {"provingJobId":"1:BASE_PARITY:83b97e35c6fcbb174b36cc01443732667146db1483c1011b6fa6834945e1aa76"} 22:16:42 [22:16:42.043] VERBOSE: prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:83b97e35c6fcbb174b36cc01443732667146db1483c1011b6fa6834945e1aa76 type=BASE_PARITY epochNumber=1 {"provingJobId":"1:BASE_PARITY:83b97e35c6fcbb174b36cc01443732667146db1483c1011b6fa6834945e1aa76","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":4} 22:16:42 [22:16:42.062] VERBOSE: prover-client:orchestrator Updating archive tree with block 1 header 0x029e217e21cd05775a4a9953c66eb1e516671fdcc2e010af72270470eaa4c9c4 22:16:42 [22:16:42.069] VERBOSE: prover-client:orchestrator Orchestrator finalised block 1 22:16:42 [22:16:42.087] INFO: prover-client:test:bb-prover-full-rollup Updating world state with new block 22:16:42 [22:16:42.091] INFO: prover-client:proving-agent Starting job id=1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 22:16:42 [22:16:42.093] INFO: prover-client:proving-agent:job-controller-33e2a110 Job controller started jobId=1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c {"jobId":"1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c"} 22:16:42 [22:16:42.184] INFO: prover-client:test:bb-prover-full-rollup Awaiting proofs 22:16:42 [22:16:42.244] INFO: bb-prover Generated witness {"circuitName":"base-parity","duration":43.01795200000015,"inputSize":160,"outputSize":96,"eventName":"circuit-witness-generation"} 22:16:46 [22:16:46.954] ERROR: bb-prover Error running operation at /tmp/b243a500/bb/tmp-g5RKfW: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:46 [22:16:46.968] ERROR: prover-client:proving-agent Job id=1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c type=BASE_PARITY failed err=Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 retry=false: Error: Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:46 at ProvingJobController.run (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:101:18) { 22:16:46 [cause]: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:46 at readProofAsFields (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/proof_utils.js:43:5) 22:16:46 at operation (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/server/bb_prover.js:315:27) 22:16:46 at runInDirectory (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/fs/run_in_dir.js:11:16) 22:16:46 at BBNativeRollupProver.createRecursiveProof (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/server/bb_prover.js:335:16) 22:16:46 at BBNativeRollupProver.getBaseParityProof (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/server/bb_prover.js:63:42) 22:16:46 at ProvingJobController.generateProof (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:164:16) 22:16:46 at ProvingJobController.run (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:96:16) { 22:16:46 generatedMessage: false, 22:16:46 code: 'ERR_ASSERTION', 22:16:46 actual: false, 22:16:46 expected: true, 22:16:46 operator: '==' 22:16:46 } 22:16:46 } 22:16:46 [22:16:46.969] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c type=BASE_PARITY totalAttempts=1 err=Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 {"provingJobId":"1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c"} 22:16:46 [22:16:46.971] INFO: prover-client:proving-agent Starting job id=1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 22:16:46 [22:16:46.972] INFO: prover-client:proving-agent:job-controller-15c0f0d3 Job controller started jobId=1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d {"jobId":"1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d"} 22:16:47 [22:16:47.000] VERBOSE: prover-client:broker-circuit-prover-facade Check for job completion notifications returned 1 job(s), snapshot ids length: 0, num outstanding jobs: 4, total jobs ready: 1 22:16:47 [22:16:47.001] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c type=BASE_PARITY: 'Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456' 22:16:47 [22:16:47.002] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456' id=1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c type=BASE_PARITY: { 22:16:47 provingJobId: '1:BASE_PARITY:a66470becf249223e48f01c5a0da18dbee1c7a3fc2f79d4a4174742d49a1c01c', 22:16:47 provingJobType: 'BASE_PARITY' 22:16:47 } 22:16:47 [22:16:47.003] VERBOSE: prover-client:broker-circuit-prover-facade Successfully retrieved 1 of 1 jobs that should be ready, total ready jobs is now: 0 22:16:47 [22:16:47.020] ERROR: prover-client:orchestrator Error thrown when proving job: Error: Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:47 at processJob (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:328:29) 22:16:47 at async Promise.all (index 0) 22:16:47 at BrokerCircuitProverFacade.retrieveJobsThatShouldBeReady (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:348:23) 22:16:47 at BrokerCircuitProverFacade.monitorForCompletedJobs (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:388:5) 22:16:47 at poll (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/promise/running-promise.js:48:21) 22:16:47 [22:16:47.021] VERBOSE: prover-client:broker-circuit-prover-facade Stopping BrokerCircuitProverFacade 22:16:47 [22:16:47.022] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d type=BASE_PARITY: { 22:16:47 provingJobId: '1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d', 22:16:47 provingJobType: 'BASE_PARITY' 22:16:47 } 22:16:47 [22:16:47.023] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=1:BASE_PARITY:f7e3fa88037136d09cfd8f15ed0db469032e74ec740a5f87facb52fd6601669e type=BASE_PARITY: { 22:16:47 provingJobId: '1:BASE_PARITY:f7e3fa88037136d09cfd8f15ed0db469032e74ec740a5f87facb52fd6601669e', 22:16:47 provingJobType: 'BASE_PARITY' 22:16:47 } 22:16:47 [22:16:47.023] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=1:BASE_PARITY:83b97e35c6fcbb174b36cc01443732667146db1483c1011b6fa6834945e1aa76 type=BASE_PARITY: { 22:16:47 provingJobId: '1:BASE_PARITY:83b97e35c6fcbb174b36cc01443732667146db1483c1011b6fa6834945e1aa76', 22:16:47 provingJobType: 'BASE_PARITY' 22:16:47 } 22:16:47 [22:16:47.024] WARN: prover-client:proving-agent:job-controller-15c0f0d3 Aborted job controller for jobId=1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d {"jobId":"1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d"} 22:16:47 [22:16:47.031] ERROR: prover-client:orchestrator Error thrown when proving job: Error: Broker facade stopped 22:16:47 at BrokerCircuitProverFacade.stop (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:200:25) 22:16:47 at TestContext.cleanup (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/mocks/test_context.ts:150:5) 22:16:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts:39:5) 22:16:47 [22:16:47.032] ERROR: prover-client:orchestrator Error thrown when proving job: Error: Broker facade stopped 22:16:47 at BrokerCircuitProverFacade.stop (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:200:25) 22:16:47 at TestContext.cleanup (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/mocks/test_context.ts:150:5) 22:16:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts:39:5) 22:16:47 [22:16:47.033] ERROR: prover-client:orchestrator Error thrown when proving job: Error: Broker facade stopped 22:16:47 at BrokerCircuitProverFacade.stop (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:200:25) 22:16:47 at TestContext.cleanup (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/mocks/test_context.ts:150:5) 22:16:47 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts:39:5) 22:16:47 [22:16:47.046] ERROR: bb-prover Error running operation at /tmp/b243a500/bb/tmp-K5YqoH: Error: Failed to generate witness: Error: ENOENT: no such file or directory, copyfile '/tmp/b243a500/acvm/tmp-US56rs/output-witness.gz' -> '/tmp/b243a500/bb/tmp-K5YqoH/partial-witness.gz' 22:16:47 [22:16:47.047] WARN: prover-client:proving-agent:job-controller-15c0f0d3 Job controller for jobId=1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d completed but job was aborted {"currentStatus":"running","jobId":"1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d"} 22:16:47 [22:16:47.047] WARN: prover-client:proving-agent Job id=1:BASE_PARITY:6e9708464d951ad3a2be29e9f2ea8c78e2057f48a644e2ad888506a4ccf1354d was aborted. Not reporting result back to broker 22:16:47 err: { 22:16:47 "type": "AbortError", 22:16:47 "message": "Proof was aborted", 22:16:47 "stack": 22:16:47 AbortError: Proof was aborted 22:16:47 at ProvingJobController.run (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:110:16) 22:16:47 "name": "AbortError" 22:16:47 } 22:16:47 FAIL src/test/bb_prover_full_rollup.test.ts (15.529 s) 22:16:47 prover/bb_prover/full-rollup 22:16:47 ✕ proves a private-only epoch with 1/1 blocks with 0/2 non-empty txs each (5824 ms) 22:16:47 ○ skipped proves all circuits 22:16:47 22:16:47 ● prover/bb_prover/full-rollup › proves a private-only epoch with 1/1 blocks with 0/2 non-empty txs each 22:16:47 22:16:47 Epoch proving failed: Error: Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:47 22:16:47 375 | const result = await this.provingPromise!; 22:16:47 376 | if (result.status === 'failure') { 22:16:47 > 377 | throw new Error(`Epoch proving failed: ${result.reason}`); 22:16:47 | ^ 22:16:47 378 | } 22:16:47 379 | 22:16:47 380 | // TODO(MW): Move this? Requires async and don't want to force root methods to be async 22:16:47 22:16:47 at TestProvingOrchestrator.finaliseEpoch (orchestrator/orchestrator.ts:377:13) 22:16:47 at test/bb_prover_full_rollup.test.ts:94:27 22:16:47 22:16:47 Test Suites: 1 failed, 1 total 22:16:47 Tests: 1 failed, 1 skipped, 2 total 22:16:47 Snapshots: 0 total 22:16:47 Time: 15.644 s 22:16:47 Ran all test suites matching prover-client/src/test/bb_prover_full_rollup.test.ts. 22:16:47 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?