Parent Log: http://ci.aztec-labs.com/0a3fab80774a5b56 Command: ef85c2b8680d9e9c yarn-project/scripts/run_test.sh prover-client/src/proving_broker/broker_prover_facade.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/9ba478709ce5c48099f79f9531da8292811cb18a Env: REF_NAME=gh-readonly-queue/staging/pr-15247-3863bc02d671057d0cffd4ced17ca317788ff190 CURRENT_VERSION=1.0.0 CI_FULL=1 Date: Tue Jun 24 14:59:25 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15247_amd64_x1-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b85794959204d424_staging 14:59:30 [14:59:30.579] INFO: prover-client:proving-broker Proving Broker started 14:59:30 [14:59:30.593] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:30 [14:59:30.694] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 14:59:30 [14:59:30.699] INFO: prover-client:proving-agent:job-controller-d5d978b1 Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:30 [14:59:30.704] INFO: prover-client:proving-agent Job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY completed outputUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22resul... 14:59:30 [14:59:30.705] INFO: prover-client:proving-broker Proving job complete id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:31 [14:59:31.599] INFO: prover-client:proving-broker Proving Broker started 14:59:31 [14:59:31.613] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:31 [14:59:31.702] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 14:59:31 [14:59:31.703] INFO: prover-client:proving-agent:job-controller-116b0334 Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:31 [14:59:31.708] INFO: prover-client:proving-agent Job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY completed outputUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22resul... 14:59:31 [14:59:31.708] INFO: prover-client:proving-broker Proving job complete id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:32 [14:59:32.607] WARN: prover-client:proving-broker Cached proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42. Not enqueuing again {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:33 [14:59:33.968] INFO: prover-client:proving-broker Proving Broker started 14:59:33 [14:59:33.971] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:34 [14:59:34.069] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 14:59:34 [14:59:34.070] INFO: prover-client:proving-agent:job-controller-6b9e021f Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:34 [14:59:34.078] ERROR: prover-client:proving-agent Job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY failed err=TEST ERROR retry=false: Error: TEST ERROR 14:59:34 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.test.ts:124:26) 14:59:34 at runNextTicks (node:internal/process/task_queues:65:5) 14:59:34 at listOnTimeout (node:internal/timers:549:9) 14:59:34 at processTimers (node:internal/timers:523:7) 14:59:34 [14:59:34.079] INFO: prover-client:proving-broker Marking proving job as failed id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 err=TEST ERROR {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:34 [14:59:34.970] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: 'TEST ERROR' 14:59:34 [14:59:34.971] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'TEST ERROR' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 14:59:34 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 14:59:34 provingJobType: 'BASE_PARITY' 14:59:34 } 14:59:34 [14:59:34.972] INFO: prover-client:broker-circuit-prover-facade Stored proof inputs for failed job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY at data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D {"id":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb","type":9,"uri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D"} 14:59:34 [14:59:34.976] WARN: prover-client:proving-broker Cached proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42. Not enqueuing again {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:35 [14:59:35.977] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: 'TEST ERROR' 14:59:35 [14:59:35.978] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'TEST ERROR' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 14:59:35 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 14:59:35 provingJobType: 'BASE_PARITY' 14:59:35 } 14:59:35 [14:59:35.978] INFO: prover-client:broker-circuit-prover-facade Stored proof inputs for failed job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY at data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D {"id":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb","type":9,"uri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D"} 14:59:35 [14:59:35.983] INFO: prover-client:proving-broker Proving Broker started 14:59:35 [14:59:35.985] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.085] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 14:59:36 [14:59:36.086] INFO: prover-client:proving-agent:job-controller-19f572a4 Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.087] INFO: prover-client:proving-broker Cancelling job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.088] INFO: prover-client:proving-broker Marking proving job as failed id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.187] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.290] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.391] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.492] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.594] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.695] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.797] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.899] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.985] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: 'Aborted' 14:59:36 [14:59:36.986] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Aborted' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 14:59:36 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 14:59:36 provingJobType: 'BASE_PARITY' 14:59:36 } 14:59:36 [14:59:36.986] WARN: prover-client:proving-agent:job-controller-19f572a4 Aborted job controller for jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.988] INFO: prover-client:proving-broker Proving Broker started 14:59:36 [14:59:36.989] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 14:59:36 [14:59:36.990] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 14:59:36 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 14:59:36 provingJobType: 'BASE_PARITY' 14:59:36 } 14:59:36 [14:59:36.991] WARN: prover-client:broker-circuit-prover-facade Running promise was not started 14:59:36 [14:59:36.991] INFO: prover-client:proving-broker Proving Broker started 14:59:37 [14:59:37.093] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 14:59:37 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 14:59:37 provingJobType: 'BASE_PARITY' 14:59:37 } 14:59:37 [14:59:37.100] WARN: prover-client:broker-circuit-prover-facade Running promise was not started 14:59:37 PASS src/proving_broker/broker_prover_facade.test.ts (10.675 s) 14:59:37 BrokerCircuitProverFacade 14:59:37 ✓ sends jobs to the broker (1023 ms) 14:59:37 ✓ handles multiple calls for the same job (2369 ms) 14:59:37 ✓ handles proof errors (2014 ms) 14:59:37 ✓ handles aborts (1005 ms) 14:59:37 ✓ rejects jobs when the facade is stopped (3 ms) 14:59:37 ✓ handles stopping while sending a proof to the broker (110 ms) 14:59:37 14:59:37 Test Suites: 1 passed, 1 total 14:59:37 Tests: 6 passed, 6 total 14:59:37 Snapshots: 0 total 14:59:37 Time: 10.759 s 14:59:37 Ran all test suites matching prover-client/src/proving_broker/broker_prover_facade.test.ts. 14:59:37 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?