Parent Log: http://ci.aztec-labs.com/61d1a335ba111d5f Command: d062a9b47b7331ac:ISOLATE=1:NAME=prover-client/src/test/bb_prover_full_rollup.test.ts:CPUS=16:MEM=96g LOG_LEVEL=verbose yarn-project/scripts/run_test.sh prover-client/src/test/bb_prover_full_rollup.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/6a9ca072a09e3c61d21e2d518f0fbf0275548eb3 Env: REF_NAME=gh-readonly-queue/next/pr-14858-96a426245731b3f48fcb6dd5e2a683c858706602 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Mon Jun 16 22:16:36 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-14858_amd64_x3-full Resources: CPU_LIST=0-127 CPUS=16 MEM=96g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_be20a80131a4ae4a_next 22:16:36 +++ id -u 22:16:36 +++ id -g 22:16:36 ++ docker run -d --name prover-client_src_test_bb_prover_full_rollup.test.ts --net=none --cpuset-cpus=0-127 --cpus=16 --memory=96g --user 1000:1000 -v/home/aztec-dev:/home/aztec-dev --mount type=tmpfs,target=/tmp,tmpfs-size=1g --workdir /home/aztec-dev/aztec-packages -e HOME -e VERBOSE -e GIT_CONFIG_GLOBAL=/home/aztec-dev/aztec-packages/build-images/src/home/.gitconfig -e FORCE_COLOR=true -e CPUS -e MEM aztecprotocol/build:3.0 /bin/bash -c 'timeout -v 600s bash -c '\''LOG_LEVEL=verbose yarn-project/scripts/run_test.sh prover-client/src/test/bb_prover_full_rollup.test.ts'\''' 22:16:37 + cid=8e7a16662bff5ebd144c5af29a8cece88dc199e3f49a415951267dd30c702273 22:16:37 + set +x 22:16:38 {"level":25,"time":1750112198812,"pid":7,"hostname":"8e7a16662bff","module":"logger","msg":"Logger initialized with level verbose"} 22:16:48 [22:16:48.170] INFO: world-state:database Creating world state data store at directory /tmp/aztec-world-state-W2kEjS/world_state with map sizes {"archiveTreeMapSizeKb":10485760,"nullifierTreeMapSizeKb":10485760,"noteHashTreeMapSizeKb":10485760,"messageTreeMapSizeKb":10485760,"publicDataTreeMapSizeKb":10485760} and 16 threads. 22:16:48 [22:16:48.256] INFO: prover-client:test:bb-prover-full-rollup Found native BB binary at /home/aztec-dev/aztec-packages/barretenberg/cpp/build/bin/bb with working directory /tmp/4f8016f2/bb 22:16:48 [22:16:48.267] INFO: prover-client:test:bb-prover-full-rollup Found native ACVM binary at /home/aztec-dev/aztec-packages/noir/noir-repo/target/release/acvm with working directory /tmp/4f8016f2/acvm 22:16:48 [22:16:48.277] INFO: bb-prover Using native BB at /home/aztec-dev/aztec-packages/barretenberg/cpp/build/bin/bb and working directory /tmp/4f8016f2/bb 22:16:48 [22:16:48.278] INFO: bb-prover Using native ACVM at /home/aztec-dev/aztec-packages/noir/noir-repo/target/release/acvm and working directory /tmp/4f8016f2/acvm 22:16:48 [22:16:48.287] INFO: prover-client:proving-broker Proving Broker started 22:16:48 [22:16:48.289] VERBOSE: prover-client:broker-circuit-prover-facade Starting BrokerCircuitProverFacade 22:16:48 [22:16:48.295] INFO: prover-client:test:bb-prover-full-rollup Proving epoch with 1/1 blocks with 0/2 non-empty txs 22:16:48 [22:16:48.296] INFO: prover-client:test:bb-prover-full-rollup Processing public functions 22:16:48 [22:16:48.301] INFO: simulator:public-processor Processed 0 successful txs and 0 failed txs in 0.00008478199999990466s {"duration":0.00008478199999990466,"rate":0,"totalPublicGas":{"daGas":0,"l2Gas":0},"totalBlockGas":{"daGas":0,"l2Gas":0},"totalSizeInBytes":0} 22:16:48 [22:16:48.331] INFO: prover-client:orchestrator Starting epoch 1 with 1 blocks 22:16:48 [22:16:48.333] INFO: prover-client:test:bb-prover-full-rollup Starting new block #1 22:16:48 [22:16:48.338] INFO: prover-client:orchestrator Starting block 1 for slot 1 22:16:48 [22:16:48.386] WARN: prover-client:orchestrator Provided no txs to orchestrator addTxs. 22:16:48 [22:16:48.386] INFO: prover-client:test:bb-prover-full-rollup Setting block as completed 22:16:48 [22:16:48.387] VERBOSE: prover-client:orchestrator Block 1 completed. Assembling header. 22:16:48 [22:16:48.392] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1 epochNumber=1 {"provingJobId":"1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1"} 22:16:48 [22:16:48.393] VERBOSE: prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1 type=BASE_PARITY epochNumber=1 {"provingJobId":"1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":1} 22:16:48 [22:16:48.395] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e epochNumber=1 {"provingJobId":"1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e"} 22:16:48 [22:16:48.397] VERBOSE: prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e type=BASE_PARITY epochNumber=1 {"provingJobId":"1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":2} 22:16:48 [22:16:48.399] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:5043657b5a868290297c8187c50bfafcf9ce00c50d5ca65c197c978ccebce1cf epochNumber=1 {"provingJobId":"1:BASE_PARITY:5043657b5a868290297c8187c50bfafcf9ce00c50d5ca65c197c978ccebce1cf"} 22:16:48 [22:16:48.403] VERBOSE: prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:5043657b5a868290297c8187c50bfafcf9ce00c50d5ca65c197c978ccebce1cf type=BASE_PARITY epochNumber=1 {"provingJobId":"1:BASE_PARITY:5043657b5a868290297c8187c50bfafcf9ce00c50d5ca65c197c978ccebce1cf","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":3} 22:16:48 [22:16:48.404] INFO: prover-client:proving-broker New proving job id=1:BASE_PARITY:4713323c4d6515c359a7f3230cd2bd0c70b94357e9ac298cd07209d0ed370488 epochNumber=1 {"provingJobId":"1:BASE_PARITY:4713323c4d6515c359a7f3230cd2bd0c70b94357e9ac298cd07209d0ed370488"} 22:16:48 [22:16:48.405] VERBOSE: prover-client:broker-circuit-prover-facade Job enqueued with broker id=1:BASE_PARITY:4713323c4d6515c359a7f3230cd2bd0c70b94357e9ac298cd07209d0ed370488 type=BASE_PARITY epochNumber=1 {"provingJobId":"1:BASE_PARITY:4713323c4d6515c359a7f3230cd2bd0c70b94357e9ac298cd07209d0ed370488","provingJobType":"BASE_PARITY","epochNumber":1,"inputsUri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input...","status":"not-found","numOutstandingJobs":4} 22:16:48 [22:16:48.407] INFO: prover-client:proving-agent Starting job id=1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1 type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 22:16:48 [22:16:48.408] INFO: prover-client:proving-agent:job-controller-e48665a4 Job controller started jobId=1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1 {"jobId":"1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1"} 22:16:48 [22:16:48.525] VERBOSE: prover-client:orchestrator Updating archive tree with block 1 header 0x087cfcccd5be41f4fab7efa3b442f733f2727dae3427276562babc324c7e16ec 22:16:48 [22:16:48.541] VERBOSE: prover-client:orchestrator Orchestrator finalised block 1 22:16:48 [22:16:48.562] INFO: prover-client:test:bb-prover-full-rollup Updating world state with new block 22:16:48 [22:16:48.624] INFO: bb-prover Generated witness {"circuitName":"base-parity","duration":52.02267100000063,"inputSize":160,"outputSize":96,"eventName":"circuit-witness-generation"} 22:16:48 [22:16:48.724] INFO: prover-client:test:bb-prover-full-rollup Awaiting proofs 22:16:53 [22:16:53.271] ERROR: bb-prover Error running operation at /tmp/4f8016f2/bb/tmp-a1hhB2: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:53 [22:16:53.283] ERROR: prover-client:proving-agent Job id=1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1 type=BASE_PARITY failed err=Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 retry=false: Error: Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:53 at ProvingJobController.run (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:101:18) { 22:16:53 [cause]: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:53 at readProofAsFields (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/proof_utils.js:43:5) 22:16:53 at operation (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/server/bb_prover.js:315:27) 22:16:53 at runInDirectory (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/fs/run_in_dir.js:11:16) 22:16:53 at BBNativeRollupProver.createRecursiveProof (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/server/bb_prover.js:335:16) 22:16:53 at BBNativeRollupProver.getBaseParityProof (/home/aztec-dev/aztec-packages/yarn-project/bb-prover/dest/prover/server/bb_prover.js:63:42) 22:16:53 at ProvingJobController.generateProof (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:164:16) 22:16:53 at ProvingJobController.run (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:96:16) { 22:16:53 generatedMessage: false, 22:16:53 code: 'ERR_ASSERTION', 22:16:53 actual: false, 22:16:53 expected: true, 22:16:53 operator: '==' 22:16:53 } 22:16:53 } 22:16:53 [22:16:53.287] INFO: prover-client:proving-broker Marking proving job as failed id=1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1 type=BASE_PARITY totalAttempts=1 err=Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 {"provingJobId":"1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1"} 22:16:53 [22:16:53.289] INFO: prover-client:proving-agent Starting job id=1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 22:16:53 [22:16:53.289] INFO: prover-client:proving-agent:job-controller-6294ff7b Job controller started jobId=1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e {"jobId":"1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e"} 22:16:53 [22:16:53.323] VERBOSE: prover-client:broker-circuit-prover-facade Check for job completion notifications returned 1 job(s), snapshot ids length: 0, num outstanding jobs: 4, total jobs ready: 1 22:16:53 [22:16:53.325] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1 type=BASE_PARITY: 'Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456' 22:16:53 [22:16:53.328] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456' id=1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1 type=BASE_PARITY: { 22:16:53 provingJobId: '1:BASE_PARITY:a522ef412bfe0c4489977348325f7f2746b9dc86785d67c118d50e1864e71ba1', 22:16:53 provingJobType: 'BASE_PARITY' 22:16:53 } 22:16:53 [22:16:53.330] VERBOSE: prover-client:broker-circuit-prover-facade Successfully retrieved 1 of 1 jobs that should be ready, total ready jobs is now: 0 22:16:53 [22:16:53.351] ERROR: prover-client:orchestrator Error thrown when proving job: Error: Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:53 at processJob (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:328:29) 22:16:53 at runNextTicks (node:internal/process/task_queues:65:5) 22:16:53 at processTimers (node:internal/timers:520:9) 22:16:53 at async Promise.all (index 0) 22:16:53 at BrokerCircuitProverFacade.retrieveJobsThatShouldBeReady (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:348:23) 22:16:53 at BrokerCircuitProverFacade.monitorForCompletedJobs (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:388:5) 22:16:53 at poll (/home/aztec-dev/aztec-packages/yarn-project/foundation/dest/promise/running-promise.js:48:21) 22:16:53 [22:16:53.354] VERBOSE: prover-client:broker-circuit-prover-facade Stopping BrokerCircuitProverFacade 22:16:53 [22:16:53.355] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e type=BASE_PARITY: { 22:16:53 provingJobId: '1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e', 22:16:53 provingJobType: 'BASE_PARITY' 22:16:53 } 22:16:53 [22:16:53.356] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=1:BASE_PARITY:5043657b5a868290297c8187c50bfafcf9ce00c50d5ca65c197c978ccebce1cf type=BASE_PARITY: { 22:16:53 provingJobId: '1:BASE_PARITY:5043657b5a868290297c8187c50bfafcf9ce00c50d5ca65c197c978ccebce1cf', 22:16:53 provingJobType: 'BASE_PARITY' 22:16:53 } 22:16:53 [22:16:53.357] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=1:BASE_PARITY:4713323c4d6515c359a7f3230cd2bd0c70b94357e9ac298cd07209d0ed370488 type=BASE_PARITY: { 22:16:53 provingJobId: '1:BASE_PARITY:4713323c4d6515c359a7f3230cd2bd0c70b94357e9ac298cd07209d0ed370488', 22:16:53 provingJobType: 'BASE_PARITY' 22:16:53 } 22:16:53 [22:16:53.360] WARN: prover-client:proving-agent:job-controller-6294ff7b Aborted job controller for jobId=1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e {"jobId":"1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e"} 22:16:53 [22:16:53.367] ERROR: prover-client:orchestrator Error thrown when proving job: Error: Broker facade stopped 22:16:53 at BrokerCircuitProverFacade.stop (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:200:25) 22:16:53 at runNextTicks (node:internal/process/task_queues:65:5) 22:16:53 at processTimers (node:internal/timers:520:9) 22:16:53 at TestContext.cleanup (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/mocks/test_context.ts:150:5) 22:16:53 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts:39:5) 22:16:53 [22:16:53.368] ERROR: prover-client:orchestrator Error thrown when proving job: Error: Broker facade stopped 22:16:53 at BrokerCircuitProverFacade.stop (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:200:25) 22:16:53 at runNextTicks (node:internal/process/task_queues:65:5) 22:16:53 at processTimers (node:internal/timers:520:9) 22:16:53 at TestContext.cleanup (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/mocks/test_context.ts:150:5) 22:16:53 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts:39:5) 22:16:53 [22:16:53.369] ERROR: prover-client:orchestrator Error thrown when proving job: Error: Broker facade stopped 22:16:53 at BrokerCircuitProverFacade.stop (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.ts:200:25) 22:16:53 at runNextTicks (node:internal/process/task_queues:65:5) 22:16:53 at processTimers (node:internal/timers:520:9) 22:16:53 at TestContext.cleanup (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/mocks/test_context.ts:150:5) 22:16:53 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/test/bb_prover_full_rollup.test.ts:39:5) 22:16:53 [22:16:53.396] INFO: bb-prover Generated witness {"circuitName":"base-parity","duration":48.260652999999365,"inputSize":160,"outputSize":96,"eventName":"circuit-witness-generation"} 22:16:53 [22:16:53.469] ERROR: bb-prover Failed to generate proof for BaseParityArtifact: Failed to generate proof. Exit code 1. Signal null. 22:16:53 [22:16:53.469] ERROR: bb-prover Error running operation at /tmp/4f8016f2/bb/tmp-g7ykJG: ProvingError: Failed to generate proof. Exit code 1. Signal null. 22:16:53 [22:16:53.470] WARN: prover-client:proving-agent:job-controller-6294ff7b Job controller for jobId=1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e completed but job was aborted {"currentStatus":"running","jobId":"1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e"} 22:16:53 [22:16:53.470] WARN: prover-client:proving-agent Job id=1:BASE_PARITY:333c5369f596233377ed8507aaf27904657b5a12799c5b8d7b904ed947b3c98e was aborted. Not reporting result back to broker 22:16:53 err: { 22:16:53 "type": "AbortError", 22:16:53 "message": "Proof was aborted", 22:16:53 "stack": 22:16:53 AbortError: Proof was aborted 22:16:53 at ProvingJobController.run (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/proving_job_controller.ts:110:16) 22:16:53 "name": "AbortError" 22:16:53 } 22:16:53 FAIL src/test/bb_prover_full_rollup.test.ts (14.896 s) 22:16:53 prover/bb_prover/full-rollup 22:16:53 ✕ proves a private-only epoch with 1/1 blocks with 0/2 non-empty txs each (5789 ms) 22:16:53 ○ skipped proves all circuits 22:16:53 22:16:53 ● prover/bb_prover/full-rollup › proves a private-only epoch with 1/1 blocks with 0/2 non-empty txs each 22:16:53 22:16:53 Epoch proving failed: Error: Unknown proving error: AssertionError [ERR_ASSERTION]: Proof length mismatch: 507 != 456 22:16:53 22:16:53 375 | const result = await this.provingPromise!; 22:16:53 376 | if (result.status === 'failure') { 22:16:53 > 377 | throw new Error(`Epoch proving failed: ${result.reason}`); 22:16:53 | ^ 22:16:53 378 | } 22:16:53 379 | 22:16:53 380 | // TODO(MW): Move this? Requires async and don't want to force root methods to be async 22:16:53 22:16:53 at TestProvingOrchestrator.finaliseEpoch (orchestrator/orchestrator.ts:377:13) 22:16:53 at test/bb_prover_full_rollup.test.ts:94:27 22:16:53 22:16:53 Test Suites: 1 failed, 1 total 22:16:53 Tests: 1 failed, 1 skipped, 2 total 22:16:53 Snapshots: 0 total 22:16:53 Time: 15.015 s 22:16:53 Ran all test suites matching prover-client/src/test/bb_prover_full_rollup.test.ts. 22:16:53 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?