ACI ·

Parent Log: http://ci.aztec-labs.com/cf66edc1d29a8480 Command: 2d72990a54a9c534 yarn-project/scripts/run_test.sh prover-client/src/proving_broker/broker_prover_facade.test.ts Commit: https://github.com/AztecProtocol/aztec-packages/commit/7edf16da6c056f9499e2dfd9c71fb9b7580f9bfc Env: REF_NAME=gh-readonly-queue/next/pr-15215-1b10c7d500621ca2e4c0db764df67f3085beb282 CURRENT_VERSION=0.87.6 CI_FULL=1 Date: Tue Jun 24 10:17:04 UTC 2025 System: ARCH=amd64 CPUS=128 MEM=493Gi HOSTNAME=pr-15215_amd64_x2-full Resources: CPU_LIST=0-127 CPUS=2 MEM=8g TIMEOUT=600s History: http://ci.aztec-labs.com/list/history_b85794959204d424_next 10:17:09 [10:17:09.267] INFO: prover-client:proving-broker Proving Broker started 10:17:09 [10:17:09.282] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:09 [10:17:09.379] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 10:17:09 [10:17:09.383] INFO: prover-client:proving-agent:job-controller-2856b5ef Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:09 [10:17:09.386] INFO: prover-client:proving-agent Job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY completed outputUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22resul... 10:17:09 [10:17:09.387] INFO: prover-client:proving-broker Proving job complete id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:10 [10:17:10.285] INFO: prover-client:proving-broker Proving Broker started 10:17:10 [10:17:10.297] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:10 [10:17:10.387] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 10:17:10 [10:17:10.388] INFO: prover-client:proving-agent:job-controller-d0857dbf Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:10 [10:17:10.391] INFO: prover-client:proving-agent Job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY completed outputUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22resul... 10:17:10 [10:17:10.392] INFO: prover-client:proving-broker Proving job complete id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:11 [10:17:11.294] WARN: prover-client:proving-broker Cached proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42. Not enqueuing again {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:12 [10:17:12.576] INFO: prover-client:proving-broker Proving Broker started 10:17:12 [10:17:12.580] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:12 [10:17:12.678] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 10:17:12 [10:17:12.678] INFO: prover-client:proving-agent:job-controller-b8fddf66 Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:12 [10:17:12.688] ERROR: prover-client:proving-agent Job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY failed err=TEST ERROR retry=false: Error: TEST ERROR 10:17:12 at Object.<anonymous> (/home/aztec-dev/aztec-packages/yarn-project/prover-client/src/proving_broker/broker_prover_facade.test.ts:124:26) 10:17:12 [10:17:12.689] INFO: prover-client:proving-broker Marking proving job as failed id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 err=TEST ERROR {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:13 [10:17:13.578] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: 'TEST ERROR' 10:17:13 [10:17:13.579] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'TEST ERROR' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 10:17:13 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 10:17:13 provingJobType: 'BASE_PARITY' 10:17:13 } 10:17:13 [10:17:13.580] INFO: prover-client:broker-circuit-prover-facade Stored proof inputs for failed job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY at data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D {"id":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb","type":9,"uri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D"} 10:17:13 [10:17:13.582] WARN: prover-client:proving-broker Cached proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42. Not enqueuing again {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:14 [10:17:14.583] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: 'TEST ERROR' 10:17:14 [10:17:14.584] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'TEST ERROR' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 10:17:14 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 10:17:14 provingJobType: 'BASE_PARITY' 10:17:14 } 10:17:14 [10:17:14.585] INFO: prover-client:broker-circuit-prover-facade Stored proof inputs for failed job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY at data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D {"id":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb","type":9,"uri":"data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22inputs%22%3A%22AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAA%3D%3D%22%7D"} 10:17:14 [10:17:14.588] INFO: prover-client:proving-broker Proving Broker started 10:17:14 [10:17:14.590] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:14 [10:17:14.690] INFO: prover-client:proving-agent Starting job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY inputsUri=data:application/json;charset=utf-8,%7B%22type%22%3A9%2C%22input... 10:17:14 [10:17:14.690] INFO: prover-client:proving-agent:job-controller-c67d0379 Job controller started jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:14 [10:17:14.692] INFO: prover-client:proving-broker Cancelling job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:14 [10:17:14.693] INFO: prover-client:proving-broker Marking proving job as failed id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY totalAttempts=1 err=Aborted {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:14 [10:17:14.791] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:14 [10:17:14.892] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:14 [10:17:14.994] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:15 [10:17:15.095] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:15 [10:17:15.196] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:15 [10:17:15.297] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:15 [10:17:15.398] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:15 [10:17:15.499] WARN: prover-client:proving-broker Proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb has already been completed {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:15 [10:17:15.589] ERROR: prover-client:broker-circuit-prover-facade Resolving proving job with error id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: 'Aborted' 10:17:15 [10:17:15.590] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Aborted' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 10:17:15 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 10:17:15 provingJobType: 'BASE_PARITY' 10:17:15 } 10:17:15 [10:17:15.590] WARN: prover-client:proving-agent:job-controller-c67d0379 Aborted job controller for jobId=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb {"jobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:15 [10:17:15.592] INFO: prover-client:proving-broker Proving Broker started 10:17:15 [10:17:15.593] INFO: prover-client:proving-broker New proving job id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb epochNumber=42 {"provingJobId":"42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb"} 10:17:15 [10:17:15.593] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 10:17:15 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 10:17:15 provingJobType: 'BASE_PARITY' 10:17:15 } 10:17:15 [10:17:15.594] WARN: prover-client:broker-circuit-prover-facade Running promise was not started 10:17:15 [10:17:15.595] INFO: prover-client:proving-broker Proving Broker started 10:17:15 [10:17:15.697] ERROR: prover-client:broker-circuit-prover-facade Job errored with 'Broker facade stopped' id=42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb type=BASE_PARITY: { 10:17:15 provingJobId: '42:BASE_PARITY:6fc11753565e9d635f3028b7106c009c3a16b7b239b7eaffb427ed8326c1c9fb', 10:17:15 provingJobType: 'BASE_PARITY' 10:17:15 } 10:17:15 [10:17:15.705] WARN: prover-client:broker-circuit-prover-facade Running promise was not started 10:17:15 PASS src/proving_broker/broker_prover_facade.test.ts (10.547 s) 10:17:15 BrokerCircuitProverFacade 10:17:15 ✓ sends jobs to the broker (1021 ms) 10:17:15 ✓ handles multiple calls for the same job (2290 ms) 10:17:15 ✓ handles proof errors (2012 ms) 10:17:15 ✓ handles aborts (1003 ms) 10:17:15 ✓ rejects jobs when the facade is stopped (3 ms) 10:17:15 ✓ handles stopping while sending a proof to the broker (111 ms) 10:17:15 10:17:15 Test Suites: 1 passed, 1 total 10:17:15 Tests: 6 passed, 6 total 10:17:15 Snapshots: 0 total 10:17:15 Time: 10.632 s 10:17:15 Ran all test suites matching prover-client/src/proving_broker/broker_prover_facade.test.ts. 10:17:15 Force exiting Jest: Have you considered using `--detectOpenHandles` to detect async operations that kept running after all tests finished?